5 ms·

Why can’t they also make glasses that translate sign language into English audio or text then?

elil17 4y ago
Translating sign languages to spoken languages is really, really hard. Sign languages have a lot of features that don't really show up in spoken languages. Let's take an example: the sign for "send email" in ASL (http://www.lifeprint.com/asl101/pages-signs/e/email.htm http://www.lifeprint.com/asl101/pages-signs/e/email.htm). If I point at you at the end of the sign it could mean "I will send you an email." If I point at myself it could mean "You should send me an email" or "Did you send me an email" depending on my facial expression. If I point off into space it could mean "I'm sending an email." If I start by pointing at you and then end the sign by pointing off into space it could mean "You should send an email." So your translator AI needs not only to understand the facial expressions and movements of the signer, but also the spacial relationships of everyone in the conversation. And that is just one aspect of the difficulty - there are many other features of sign languages that are just as hard to translate. Perhaps this is the sort of thing that future AI systems could do. But it is quite complex.
dymk 4y ago
I'm sure that's something someone could do! And it sounds like a very fun project. It probably hasn't already been done because there already exist very good speech-to-text models and not many (any?) sign-language-to-{text,audio} models
zie 4y ago
Sign language is a complex language, so something has to learn yet another language. Perhaps made all the more complex as it's spatial and visual instead of verbal or written. Not every deaf person uses sign language and there are many different sign languages in the world. American Sign Language(ASL) is but one of these languages.
- richbell 4y ago
  > Sign language is a complex language Specifically, sign languages are not visual representations of existing languages (e.g. ASL and English) but completely different languages altogether.
  - SamuelAdams 4y ago
    And even then, there are different dialects of ASL.
  - zie 4y ago
    Exactly. Though Visual "representations"(for lack of a better word) of English do exist in dialects like SEE(Signed Exact English), which is obviously not the same as ASL. I'm sure other signed languages have rough equivalents in their regional areas as well. I know Mexico has a few different signed languages, though I only have passing familiarity with 1 of their signed languages and it's definitely not a representation of Spanish.
  - Daub 4y ago
    I was waiting for someone to say this. I have taught Humanities in a program which included a significant number of deaf students. Communication was enabled via translators. The difference in language culture plus the fact that everything was mediated through a third party made the whole experience profoundly unsatisfactory. Nuance was completely lost. Perhaps if I had recovered training in sign language things world have been different.
dgunay 4y ago
Not an expert in either technology but I would imagine a lot of the difficulties of speech to text (bad recording conditions, variety of accents & pronunciation differences, etc) also have analogues in computer recognition of hand signs (camera alignment is bad, cut off, lighting is bad, someone's hand signs are lazily performed or slightly different than textbook ASL, etc). Speech to text was "solved" a long time ago but I've seen it take many years to become as usable as it has recently. And it still regularly is frustrating to use for me!
- tsimionescu 4y ago
  My experience of speech to text has been terrible. As soon as someone has a bit of an accent, it becomes completely incoherent, even if all participants understand the accent perfectly. Even for good accents, using any kind of technical terms or proper names throws it off. And even when none of these are a problem, it still has at least a few percent error, even for English.
  - baseline-shift 4y ago
    true. But all transcription tools are terrible transcribing technical language if someone has accented English.
jmcmaster 4y ago
Microsoft Research had early implementation for this kind of visual sign language translation with Kinect & ASL recognition in 2013 or so. I expect that with the death of Kinect in the market it stayed lab-bound.
quantumquetzal 4y ago
Sign language (speaking with ASL in mind, unfamiliar with others) uses a different grammatical structure from American English. To boot, a signer can adopt different body languages to imbue their message with different tonality. Current ML models (as far as I’m aware) struggle with capturing that tonality in speech-to-text, and there’s _far_ less data for sign language recognition, regrettably.

Translating sign languages to spoken languages is really, really hard. Sign languages have a lot of features that don't really show up in spoken languages. Let's take an example: the sign for "send email" in ASL (http://www.lifeprint.com/asl101/pages-signs/e/email.htm http://www.lifeprint.com/asl101/pages-signs/e/email.htm). If I point at you at the end of the sign it could mean "I will send you an email." If I point at myself it could mean "You should send me an email" or "Did you send me an email" depending on my facial expression. If I point off into space it could mean "I'm sending an email." If I start by pointing at you and then end the sign by pointing off into space it could mean "You should send an email." So your translator AI needs not only to understand the facial expressions and movements of the signer, but also the spacial relationships of everyone in the conversation. And that is just one aspect of the difficulty - there are many other features of sign languages that are just as hard to translate. Perhaps this is the sort of thing that future AI systems could do. But it is quite complex.

dymk 4y ago

I'm sure that's something someone could do! And it sounds like a very fun project. It probably hasn't already been done because there already exist very good speech-to-text models and not many (any?) sign-language-to-{text,audio} models

zie 4y ago

Sign language is a complex language, so something has to learn yet another language. Perhaps made all the more complex as it's spatial and visual instead of verbal or written. Not every deaf person uses sign language and there are many different sign languages in the world. American Sign Language(ASL) is but one of these languages.

richbell 4y ago
> Sign language is a complex language Specifically, sign languages are not visual representations of existing languages (e.g. ASL and English) but completely different languages altogether.
- SamuelAdams 4y ago
  And even then, there are different dialects of ASL.
- zie 4y ago
  Exactly. Though Visual "representations"(for lack of a better word) of English do exist in dialects like SEE(Signed Exact English), which is obviously not the same as ASL. I'm sure other signed languages have rough equivalents in their regional areas as well. I know Mexico has a few different signed languages, though I only have passing familiarity with 1 of their signed languages and it's definitely not a representation of Spanish.
- Daub 4y ago
  I was waiting for someone to say this. I have taught Humanities in a program which included a significant number of deaf students. Communication was enabled via translators. The difference in language culture plus the fact that everything was mediated through a third party made the whole experience profoundly unsatisfactory. Nuance was completely lost. Perhaps if I had recovered training in sign language things world have been different.

dgunay 4y ago

Not an expert in either technology but I would imagine a lot of the difficulties of speech to text (bad recording conditions, variety of accents & pronunciation differences, etc) also have analogues in computer recognition of hand signs (camera alignment is bad, cut off, lighting is bad, someone's hand signs are lazily performed or slightly different than textbook ASL, etc). Speech to text was "solved" a long time ago but I've seen it take many years to become as usable as it has recently. And it still regularly is frustrating to use for me!

tsimionescu 4y ago
My experience of speech to text has been terrible. As soon as someone has a bit of an accent, it becomes completely incoherent, even if all participants understand the accent perfectly. Even for good accents, using any kind of technical terms or proper names throws it off. And even when none of these are a problem, it still has at least a few percent error, even for English.
- baseline-shift 4y ago
  true. But all transcription tools are terrible transcribing technical language if someone has accented English.

jmcmaster 4y ago

Microsoft Research had early implementation for this kind of visual sign language translation with Kinect & ASL recognition in 2013 or so. I expect that with the death of Kinect in the market it stayed lab-bound.

quantumquetzal 4y ago

Sign language (speaking with ASL in mind, unfamiliar with others) uses a different grammatical structure from American English. To boot, a signer can adopt different body languages to imbue their message with different tonality. Current ML models (as far as I’m aware) struggle with capturing that tonality in speech-to-text, and there’s _far_ less data for sign language recognition, regrettably.