3 ms·

ASL student here. I just wanted to highlight how many challenges there are to translating, even for humans. Here's two examples. First is dialect. One would th

ASL student here. I just wanted to highlight how many challenges there are to translating, even for humans. Here's two examples. First is dialect. One would think the Internet/youtube/videochat would vaporize geography differences, and it's starting to happen, but that's not (yet) the case. Lots of signs have "synonyms" based on signer's preference, etymology, who they learned from and who they hang out with, etc. Another is grammar. What OP is doing in his video, for example is English grammar, where signs are directly substitited for words. But ASL has its own sign order, modifiers like facial expressions, and idioms for brevity. For example "have you ever been to San Francisco before?" might be "SF TOUCH YOU FINISH" with raised eyebrows at the end, as a modifier. Note also there are no conjugations or articles like some languages, but there are pronouns. In fact, there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation. I think with large enough training sets, this will all be mitigated, like Google needed years of speech samples to get Translate working okay.

mabbo 8y ago
I took 2.5 years of ASL in University, and I've always been annoyed by this, but had a hard time describing what I mean to others. You have just summed up the problem perfectly in a few sentences, and I thank you for that. That said, I think the OP is still on the right track here: > I put it together so you can train it on your own set of word and sign/gesture combos. A Deaf person should be able to train the system on each command they want so that it works for them automatically in the dialect they want. > there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation My very favourite thing about ASL, having learned programming beforehand, was that I could assign people to variables/registers in space. "John point at spot to my left said to Susan spot on right bla bla bla" and then later be able to just point at my left and everyone knows I mean John. And then if John moves to the right, I've just moved him into that register and can reassign the left one to someone or something else. And with a group of experienced Signers, everyone just comprehends this perfectly.
- imglorp 8y ago
  Ah, pointing at sign space for someone... I like to think of that like a pronoun (there's probably an official term). There's also this thing where you introduce a character, then make up a "temporary name sign" on the fly. Don't know what that's called either :)
  - mercurywells 8y ago
    The term is 'deixis'. Not sure about the latter.
- mncharity 8y ago
  > ASL [...] programming Hand pose tracking is becoming increasingly viable, both with rapidly advancing ML of camera video[1], and perhaps with VR gloves. So as someone interested in expert UIs for programming inside VR/AR, I ask... Any suggestions for ASL linguistics resources to mine for non-novice UI idioms/vocabularies/grammars/etc? Your "3-space as namespace" as one example. Gaze tracking will similarly be available. Facial expression, at least while wearing HMDs, regrettably not so much (despite prototypes). Is there signing experience with hand-held objects? - fast fine-control "fiddling with a pencil" is available with sub-millimeter few-degrees precision 6DOF, but it's unclear what vocabulary/grammar to use with it. Big picture: VR/AR seems an opportunity to leverage accumulated insights from signing. [1] eg, https://github.com/xinghaochen/awesome-hand-pose-estimation https://github.com/xinghaochen/awesome-hand-pose-estimation
  - imglorp 8y ago
    Student, not expert, but.. the definitive reference is [1]. The front matter establishes a system for taxonomy of signs: one hand or two, handshapes, location, motion, facial expression, modifiers, classifiers, etc. and then has a big dictionary. That front matter should be very helpful for translation authors. Note however, that's their dialect/accent, and other regions will have variations. Bill Vicars [2] is pretty good about cataloging many variations but it's all video material, not reference. 1. https://www.amazon.com/Gallaudet-Dictionary-American-Sign-Language/dp/1563682826 https://www.amazon.com/Gallaudet-Dictionary-American-Sign-La... 2. http://asluniversity.com http://asluniversity.com
pbhjpbhj 8y ago
As you intimate, it's a very common misunderstanding that sign languages are forms of spoken languages rather than languages in their own right. This page (https://signly.co/ https://signly.co/, for a sign app that gives prerecorded interpretation of public info) has an example of BSL vs English grammar - see "Show Example". It appears the user is doing an ASL form of Signed English.
CobrastanJorji 8y ago
A part of me wishes that written sign languages caught on. With a large enough corpus, they'd be way easier to translate into other languages, and it'd also be easier to translate between signing and written sign language. The way things stand, ML solutions need to perform both steps of the translation, which is so much harder.
- imglorp 8y ago
  That would be interesting. It seems there's going to be a tradeoff between entropy (information content) in a human sign versus the written form. An example might be the sign RAIN, which could mean anything from drizzle to monsoon depending on how energetic and exagerated the motion. Differentiating which would would need some context, baselining how animated the signer is in the conversation. If you wrote that down you might need RAIN... for one and RAIN!!!!! for the other.

mabbo 8y ago

I took 2.5 years of ASL in University, and I've always been annoyed by this, but had a hard time describing what I mean to others. You have just summed up the problem perfectly in a few sentences, and I thank you for that. That said, I think the OP is still on the right track here: > I put it together so you can train it on your own set of word and sign/gesture combos. A Deaf person should be able to train the system on each command they want so that it works for them automatically in the dialect they want. > there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation My very favourite thing about ASL, having learned programming beforehand, was that I could assign people to variables/registers in space. "John point at spot to my left said to Susan spot on right bla bla bla" and then later be able to just point at my left and everyone knows I mean John. And then if John moves to the right, I've just moved him into that register and can reassign the left one to someone or something else. And with a group of experienced Signers, everyone just comprehends this perfectly.

imglorp 8y ago
Ah, pointing at sign space for someone... I like to think of that like a pronoun (there's probably an official term). There's also this thing where you introduce a character, then make up a "temporary name sign" on the fly. Don't know what that's called either :)
- mercurywells 8y ago
  The term is 'deixis'. Not sure about the latter.
mncharity 8y ago
> ASL [...] programming Hand pose tracking is becoming increasingly viable, both with rapidly advancing ML of camera video[1], and perhaps with VR gloves. So as someone interested in expert UIs for programming inside VR/AR, I ask... Any suggestions for ASL linguistics resources to mine for non-novice UI idioms/vocabularies/grammars/etc? Your "3-space as namespace" as one example. Gaze tracking will similarly be available. Facial expression, at least while wearing HMDs, regrettably not so much (despite prototypes). Is there signing experience with hand-held objects? - fast fine-control "fiddling with a pencil" is available with sub-millimeter few-degrees precision 6DOF, but it's unclear what vocabulary/grammar to use with it. Big picture: VR/AR seems an opportunity to leverage accumulated insights from signing. [1] eg, https://github.com/xinghaochen/awesome-hand-pose-estimation https://github.com/xinghaochen/awesome-hand-pose-estimation
- imglorp 8y ago
  Student, not expert, but.. the definitive reference is [1]. The front matter establishes a system for taxonomy of signs: one hand or two, handshapes, location, motion, facial expression, modifiers, classifiers, etc. and then has a big dictionary. That front matter should be very helpful for translation authors. Note however, that's their dialect/accent, and other regions will have variations. Bill Vicars [2] is pretty good about cataloging many variations but it's all video material, not reference. 1. https://www.amazon.com/Gallaudet-Dictionary-American-Sign-Language/dp/1563682826 https://www.amazon.com/Gallaudet-Dictionary-American-Sign-La... 2. http://asluniversity.com http://asluniversity.com

pbhjpbhj 8y ago

As you intimate, it's a very common misunderstanding that sign languages are forms of spoken languages rather than languages in their own right. This page (https://signly.co/ https://signly.co/, for a sign app that gives prerecorded interpretation of public info) has an example of BSL vs English grammar - see "Show Example". It appears the user is doing an ASL form of Signed English.

CobrastanJorji 8y ago

A part of me wishes that written sign languages caught on. With a large enough corpus, they'd be way easier to translate into other languages, and it'd also be easier to translate between signing and written sign language. The way things stand, ML solutions need to perform both steps of the translation, which is so much harder.

imglorp 8y ago
That would be interesting. It seems there's going to be a tradeoff between entropy (information content) in a human sign versus the written form. An example might be the sign RAIN, which could mean anything from drizzle to monsoon depending on how energetic and exagerated the motion. Differentiating which would would need some context, baselining how animated the signer is in the conversation. If you wrote that down you might need RAIN... for one and RAIN!!!!! for the other.