8 ms·

Learn American Sign Language Fingerspelling with Machine Learning

81 points by joe5150 5y ago

Johnover_Board 5y ago
Thanks a bunch for this! I've been meaning to learn ASL but it doesn't usually get fun enough to engage long term online with video lectures.
Ajedi32 5y ago
Would love to see something like this implemented a keyboard replacement. Would be a neat way to passively learn ASL while doing other tasks.
- gugagore 5y ago
  Learning the manual alphabet is not learning ASL. Most people can become pretty comfortable with producing finger spelling in a matter of a week or two. There's a ton more to learn beyond finger spelling, not to mention that receptive finger spelling is a challenging skill even for those who have learned ASL for many years.
SatvikBeri 5y ago
This is great! I've tried to use Anki cards to remember fingerspelling after my first ASL class, but it's pretty hard due to the lack of detailed feedback – this is a lot better.
adzm 5y ago
Really wish this was available on a mobile device. Would be a great way to practice while I'm waiting on something.
- yorwba 5y ago
  It works just fine in Firefox for Android for me after enabling desktop mode.
tomcooks 5y ago
A really dumb question, which a close friend that teaches sign language in Europe couldn't answer to: why do we have different sign languages, one per spoken language, instead of a common international single language? I understand there's partly nationalism in this, and that if Esperanto taught us something is that international languages are HARD, but I can't help but think that a single standard sign language would help communication between users and easier involvement of speakers. Again, I hope I don't sound offensive, I'm just curious.
- InvisibleUp 5y ago
  A lot of these sign languages were developed before the Internet or any easy mass communication via video. Each region developed their own language, and at this point it's so ingrained into local cultures that trying to get every sign language user to switch to some international language would be like trying to get everyone to switch to Esperanto. It's not happening.
- eipipuz 5y ago
  (disclaimer: I'm not Deaf, just interested in ASL.) Signs very geographically even in ASL. (See: https://www.reddit.com/r/asl/comments/dcvqo9/is_there_much_difference_between_west_coast_and/ https://www.reddit.com/r/asl/comments/dcvqo9/is_there_much_d...) Wouldn't your argument hold for spoken languages as well? Why do we have french and spanish? Because it would be impossible to force most people to learn the other language. English is the widely spoken for economical reasons and yet that's only 13% of the world population. No sign language has that kind of pull. We would need TV shows / movies that pushed a particular sign. The internet helps but how do you prevent groups from drifting? At some point accents become dialect that become languages. Written ASL doesn't seem to be growing. Why do people communicate in different languages? Because they live in slightly disconnected social circles.
- CatsEyes 5y ago
  American Sign Language is rooted in French Sign Language, because a teacher at a long-running school for the deaf there came to America to teach. British Sign Language developed in a different direction, where fingerspelling the alphabet is very different from how it is done in ASL. So even though US and England share a spoken language, signed expression developed differently. Same deal with signed languages around the world. They generally develop locally and incorporate influences from elsewhere, much as spoken languages do, with the local dominant spoken language having the most influence.
eipipuz 5y ago
If you are interested in ASL, I think a good resource is: https://www.lifeprint.com/ https://www.lifeprint.com/
- krick 5y ago
  Somewhat offtop, but I wanted to ask: how much universal intelligibility there is in sign languages? I understand that sign languages are, well, languages, and thus not readily inter-intelligible. But so are, perhaps, our native languages, yet here we are speaking English. And not just because this is a somewhat SV forum: english is just internationally popular right now. Of course, most people I meet don't really speak english, and lingua franca will change based on the region (at many places most people will know nothing but local language), but learning english is still way more practical worldwide (outside of USA and UK) than learning estonian, and speaking english I have good chances to be understood in most big cities in Europe and worldwide. Also, I understand that it's hard to compare, because most people don't speak any sign languages, but is there something similar with sign languages? Honestly, I feel weird desire to learn some sign language, but there's absolutely zero practical reasons for me to do so, and I wonder if learning, say, ASL, could help me interact with deaf people in my country or somewhere else. Sure, I won't understand them when they speak their local dialect, but is there something common that most deaf people would probably understand?
  - cloudpanda 5y ago
    There is an artificial universal sign language used at deaf conventions. It isn’t particularly popular or well known (to fluency) though. Most deaf people that know separate sign systems or sign languages will figure out ways to communicate across those language barriers at extreme speed. They are unmatchable charade experts. I have seen deaf people from different continents have entire conversations in pidgin they bridge together over a few minutes. ASL is perhaps more universal than English in one way. Most missionaries that established ASL Brought it with them as their primary teaching tool to deaf children. So from Jamaica to Ghanna, Canada, many places in Africa, there will be many locations that sign a local variant of ASL. Perhaps maintaining 50-60% of the parent sign language. ASL’s parent sign language is French, it is one of the older sign language and so has maintained less of its French roots, but we still use French twos and threes in our signed numbering system.
dwighttk 5y ago
seems to be having trouble identifying M, N, D, and K for me... the hand skeleton is identifying the shape my hand is making but I have to rotate around a few different axes for a while until it decides to accept it.
laszlokorte 5y ago
works really well, even in a dark room in which I could barely see my own hand on the webcam. I am really impressed.
gugagore 5y ago
It is cool to see handtracking working in my browser! Unsurprisingly, there are serious flaws with this tool. It's hard to get this right, and there are endless "learn ASL" projects that only ever get as far as recognizing canonical handshapes. The first word I was prompted to fingerspell was ABLE. Instead of B, I signed 4 (fingers spread) and it accepted it. Just a curiosity. The second word was LOVE. The E in LOVE, for many users, will be different than a "textbook" E, because of assimilation. The preceding V affects the E (only 2 fingers are selected instead of all 4). There is a perfect explanation here: https://www.lifeprint.com/asl101/pages-layout/handshapes.htm https://www.lifeprint.com/asl101/pages-layout/handshapes.htm
xvilka 5y ago
I recently started to learn ASL on Memrise, they have good beginner courses[1]. [1] https://app.memrise.com/courses/english/american-sign-language-asl/ https://app.memrise.com/courses/english/american-sign-langua...
junon 5y ago
This is cool! Few things: Add the words that were already spelled to a `Set` and check for them again when making new word choices. I got "NUMB" three times in a single session. Z's tracking just flat out doesn't work. Q is signed different than I learned it, apparently. I sign Q as an upside down P, but the tracking apparently wants an upside down G. I don't doubt people sign it that way but as others have mentioned this is going to be different for different people. Lastly, I had problems with going too fast. It really felt like I had to strain myself to get it to recognize. More tolerances in the recognition would be nice.
bussiere 5y ago
If you made a website to learn American sign langage with this system, i will be ready to pay for it :) Thanks btw

81 points by joe5150 5y ago

Thanks a bunch for this! I've been meaning to learn ASL but it doesn't usually get fun enough to engage long term online with video lectures.

Ajedi32 5y ago

Would love to see something like this implemented a keyboard replacement. Would be a neat way to passively learn ASL while doing other tasks.

gugagore 5y ago
Learning the manual alphabet is not learning ASL. Most people can become pretty comfortable with producing finger spelling in a matter of a week or two. There's a ton more to learn beyond finger spelling, not to mention that receptive finger spelling is a challenging skill even for those who have learned ASL for many years.

SatvikBeri 5y ago

This is great! I've tried to use Anki cards to remember fingerspelling after my first ASL class, but it's pretty hard due to the lack of detailed feedback – this is a lot better.

adzm 5y ago

Really wish this was available on a mobile device. Would be a great way to practice while I'm waiting on something.

yorwba 5y ago
It works just fine in Firefox for Android for me after enabling desktop mode.

tomcooks 5y ago

A really dumb question, which a close friend that teaches sign language in Europe couldn't answer to: why do we have different sign languages, one per spoken language, instead of a common international single language? I understand there's partly nationalism in this, and that if Esperanto taught us something is that international languages are HARD, but I can't help but think that a single standard sign language would help communication between users and easier involvement of speakers. Again, I hope I don't sound offensive, I'm just curious.

InvisibleUp 5y ago
A lot of these sign languages were developed before the Internet or any easy mass communication via video. Each region developed their own language, and at this point it's so ingrained into local cultures that trying to get every sign language user to switch to some international language would be like trying to get everyone to switch to Esperanto. It's not happening.
eipipuz 5y ago
(disclaimer: I'm not Deaf, just interested in ASL.) Signs very geographically even in ASL. (See: https://www.reddit.com/r/asl/comments/dcvqo9/is_there_much_difference_between_west_coast_and/ https://www.reddit.com/r/asl/comments/dcvqo9/is_there_much_d...) Wouldn't your argument hold for spoken languages as well? Why do we have french and spanish? Because it would be impossible to force most people to learn the other language. English is the widely spoken for economical reasons and yet that's only 13% of the world population. No sign language has that kind of pull. We would need TV shows / movies that pushed a particular sign. The internet helps but how do you prevent groups from drifting? At some point accents become dialect that become languages. Written ASL doesn't seem to be growing. Why do people communicate in different languages? Because they live in slightly disconnected social circles.
CatsEyes 5y ago
American Sign Language is rooted in French Sign Language, because a teacher at a long-running school for the deaf there came to America to teach. British Sign Language developed in a different direction, where fingerspelling the alphabet is very different from how it is done in ASL. So even though US and England share a spoken language, signed expression developed differently. Same deal with signed languages around the world. They generally develop locally and incorporate influences from elsewhere, much as spoken languages do, with the local dominant spoken language having the most influence.

eipipuz 5y ago

If you are interested in ASL, I think a good resource is: https://www.lifeprint.com/ https://www.lifeprint.com/

krick 5y ago
Somewhat offtop, but I wanted to ask: how much universal intelligibility there is in sign languages? I understand that sign languages are, well, languages, and thus not readily inter-intelligible. But so are, perhaps, our native languages, yet here we are speaking English. And not just because this is a somewhat SV forum: english is just internationally popular right now. Of course, most people I meet don't really speak english, and lingua franca will change based on the region (at many places most people will know nothing but local language), but learning english is still way more practical worldwide (outside of USA and UK) than learning estonian, and speaking english I have good chances to be understood in most big cities in Europe and worldwide. Also, I understand that it's hard to compare, because most people don't speak any sign languages, but is there something similar with sign languages? Honestly, I feel weird desire to learn some sign language, but there's absolutely zero practical reasons for me to do so, and I wonder if learning, say, ASL, could help me interact with deaf people in my country or somewhere else. Sure, I won't understand them when they speak their local dialect, but is there something common that most deaf people would probably understand?
- cloudpanda 5y ago
  There is an artificial universal sign language used at deaf conventions. It isn’t particularly popular or well known (to fluency) though. Most deaf people that know separate sign systems or sign languages will figure out ways to communicate across those language barriers at extreme speed. They are unmatchable charade experts. I have seen deaf people from different continents have entire conversations in pidgin they bridge together over a few minutes. ASL is perhaps more universal than English in one way. Most missionaries that established ASL Brought it with them as their primary teaching tool to deaf children. So from Jamaica to Ghanna, Canada, many places in Africa, there will be many locations that sign a local variant of ASL. Perhaps maintaining 50-60% of the parent sign language. ASL’s parent sign language is French, it is one of the older sign language and so has maintained less of its French roots, but we still use French twos and threes in our signed numbering system.

dwighttk 5y ago

seems to be having trouble identifying M, N, D, and K for me... the hand skeleton is identifying the shape my hand is making but I have to rotate around a few different axes for a while until it decides to accept it.

laszlokorte 5y ago

works really well, even in a dark room in which I could barely see my own hand on the webcam. I am really impressed.

gugagore 5y ago

It is cool to see handtracking working in my browser! Unsurprisingly, there are serious flaws with this tool. It's hard to get this right, and there are endless "learn ASL" projects that only ever get as far as recognizing canonical handshapes. The first word I was prompted to fingerspell was ABLE. Instead of B, I signed 4 (fingers spread) and it accepted it. Just a curiosity. The second word was LOVE. The E in LOVE, for many users, will be different than a "textbook" E, because of assimilation. The preceding V affects the E (only 2 fingers are selected instead of all 4). There is a perfect explanation here: https://www.lifeprint.com/asl101/pages-layout/handshapes.htm https://www.lifeprint.com/asl101/pages-layout/handshapes.htm

xvilka 5y ago

I recently started to learn ASL on Memrise, they have good beginner courses[1]. [1] https://app.memrise.com/courses/english/american-sign-language-asl/ https://app.memrise.com/courses/english/american-sign-langua...

junon 5y ago

This is cool! Few things: Add the words that were already spelled to a `Set` and check for them again when making new word choices. I got "NUMB" three times in a single session. Z's tracking just flat out doesn't work. Q is signed different than I learned it, apparently. I sign Q as an upside down P, but the tracking apparently wants an upside down G. I don't doubt people sign it that way but as others have mentioned this is going to be different for different people. Lastly, I had problems with going too fast. It really felt like I had to strain myself to get it to recognize. More tolerances in the recognition would be nice.

bussiere 5y ago

If you made a website to learn American sign langage with this system, i will be ready to pay for it :) Thanks btw