2 ms·
I may have written too much. TLDR: This is typing english letters with hand gestures for each letter, not sign language translation which is HARD. There have
by lsiebert 8y ago
I may have written too much. TLDR: This is typing english letters with hand gestures for each letter, not sign language translation which is HARD.
There have been devices like this since the 80s, because it basically does hand shapes, not signs (which have movement, grammar, spatial references, body placement references and other aspects that a hand and wrist device won't capture).
This looks suitable for fingerspelling only (Think, instead of words, spelling out everything you want to say using the letters of the alphabet, and you'll get why this isn't a full solution, though you should understand that sign languages are discrete languages with their own grammar and syntax not just english words as signs, so... I guess imagine spelling out what you want to say but it's transliterated french and using latin declensions or something).
Better then nothing, sure, but hardly translating sign language.
Can you do it with a camera and machine learning? I very much doubt it. I am not sure current gesture segmentation methods could pick out discrete signs or even hand shapes in different orientations, given that there isn't a neutral state one returns between "words".
Then you need to do gesture classification, differentiating based on handshapes (which can transform during a sign, with the nature of the transition, as well as it's speed, the size of the over all gesture, incorporated facial language etc is meaningful, and can be modified by previous or future signs, spatial designations etc), and THEN you need to do stateful real time machine language translation from the decoded signs into English (or Swahili possibly), because a nations sign language isn't english (or whatever your local language is) with the words signed, it's it's own thing, and there may not be a one to one correspondence between words.
In ASL (and I can't speak to other sign languages), you often put the subject of the sentence first (IE instead of saying, "who's the new guy in the purple tie?" the grammar is more like "GUY THERE, WEARING TIE PURPLE, NEW, WHO?") And that's something simple, where gender is included explicitly, no past or future tenses etc. Of course there's also code switching, and formalized systems for conveying English like Signing Exact English.
Oh, did I mention that no sign language has a written form? Linguists and others can create code designations based on what they see as sign word correspondences, but the same sign can be modified, such that the sign you use for a pretty face can be modified to indicate degree of beauty, so the sign for gorgeous and the sign for pretty need to be differentiated, by software.
Oh and if you got it working for one country... you got it working for one country. British sign language and american sign language are radically different, and basically every country has it's own sign language, though they may have some shared roots. Even in the US, because of segregated schools for the deaf, there's a black ASL dialect.
Anyway, we might have the technology at this point to do sign language translation, but probably not, but even if we did, the funding to get it from working on set sign language phrases in an MIT or Carnegie Melon Lab to something that can actually translate sign language isn't there.
If someone wanted to start camera classification for gestures, I'd start with recognizing flipping the bird fast enough to blur it in a live video feed, because people might actually pay for that.