12 ms·
SignAloud: Gloves That Translate Sign Language into Text and Speech
- skrap 11y agoTo properly translate ASL, you need to pick up a significant amount of grammar from the face (called non-manual markers) as well as body position, 3d capture of spatial relationships between signs as they are made, and other features which would be inaccessible to these sensors. And then there's the fact that ASL isn't serialized English (it's actually unrelated to English at all, and has its roots in the French Sign Language). There's no one-to-one mapping between ASL signs and English words, just as there's no one-to-one mapping between Chinese and French, or any two languages. And ASL has grammar features not seen in any spoken language, to complicate things further. These sensors could perhaps be used to translate other sign languages, but I'm really skeptical.
- emodendroket 11y agoYeah, even if they could perfectly transcribe the signs you'd have real problems/limitations here I think.
- gugagore 11y agoIt's not really clear what it means to transcribe the signs anyway. There isn't very great notation for signed languages
- Zuider 11y agoI wasn't aware of the existence of any notation at all! The dictionary I have uses photographs, other dictionaries and textbooks use drawings of people speaking SL. The nearest thing to a notation in this case consists of a verbal description of face and hand gestures as they are carried out in real time. I have often wondered if it would be possible to create a notation using the simplest and most abstract rendition of an actual SL speaker's signing. Lines and curves that hold the semantic essence of the movement of face, hand and posture. It would take a native speaker to be able to abstract what is truly essential to meaning, and a community to agree on its elegance and clarity. Off topic, but connected: I discovered from experience that small children can form the SL alphabet with their hands before they acquire the coordination to draw written letters. I found it very useful in teaching my children to read.
- Zuider 11y agoCorrection to my comment above: On taking out my Dictionary of British Sign Language, I find that there is indeed an intricate notation used throughout.
- emodendroket 11y agoDoes it matter? As long as it is consistent and we can reliably translate it (admittedly those are big "ifs") who cares if it's standard?
- gugagore 11y agoYes it matters. I didn't say anything about a standard. I don't mean that "there isn't an agreed upon notation that Deaf Americans use to write ASL", I mean "we don't have a good way represent signs that allows us to search with that representation, apply phonological rules, and synthesize signs from that representation". We have all sorts of representations for spoken speech---a waveform, spectrogram, vocal tract models, IPA--and these all get used in a speech recognition (or synthesis) system. For ASL we have movies, still-image cartoons that depict motion, categories for some parameters like handshape, some linguistic insights, and I'm not sure what else.
- luxpir 11y agoVery cool, but it's not quite the revolution described by the inventors in the video - isn't it only one-way, assuming a lot of the deaf person's lip-reading ability?
- emodendroket 11y agoIf a deaf person can read lips they can probably also speak English.
- luxpir 11y agoI'm not sure we understand each other. If the signing person can speak to employers etc. now via the device, they can only speak back with their mouths, or possibly speech recognition if well set up. The deaf person would have to be able to lip-read people who potentially don't know how to help them in that area. In other words, it might still be tricky to communicate.
- emodendroket 11y agoYou are right; I totally misunderstood you. Sorry.
- ohitsdom 11y agoSeems cool, but I wonder how much of this is faked in the demo video. The man signs "T" and it's translated to "Thomas", "N" to "Navid", and "L" to "Lemelson". So unless this is a learning system as well, I'm skeptical. Lot of potential here for sure though, hope they keep it up.
- ToastyMallows 11y agoI wonder if they've just hard coded their names for now because they had trouble doing fingerspelling, because I doubt that someone's name-sign is just a 'T'. That's really close to the sign for 'bathroom', which would be weird :)
- pavel_lishin 11y agoCould be a home-sign, or a common shortcut. If two people are discussing two people who aren't physically present, using just their first initials could be a very obvious way of referring to them unambiguously without having to sign their full names.
- pvinis 11y agosome friends of mine translate sign language to text/speech using leap motion and/or kinect to figure out the best way for it. there is a startup that uses leap motion and an ipad but it looks very clunky to use. my friends are connecting to the user's smartphone for the translation work.
- buraksarica 11y agoAnother example: http://www.turtela.com/en/main http://www.turtela.com/en/main
- zardeh 11y agoI always find these cool, and then remember that I know someone who built a working (well working!) prototype of this in a weekend no almost no fanfare, from practically scratch.
- avs733 11y agowith the caveat of not knowing your that someone in any way shape or form... These are students, young/new engineers. This is impressive and should be lauded as should all attempts to push what an individual person is capable of. It may not be new or novel to everyone, but it is to them...basically, cut them a bit of slack.
- zardeh 11y agoThe people I'm talking about were also students, (University Hackathon). A group of four, I want to say 2 sophomores and 2 freshmen. This was 2-3 years ago, so machine learning wasn't advanced (and even now, getting a functional NN running and trained in a weekend isn't feasible) so they only had it recognize a small set of commands, but it could do that.
- nowlnowl 11y agoPlease link them here so we together can discuss it and hopefully make some fanfare!
- fsiefken 11y agoSo now you can text entry without using a keyboard or your voice, good for rsi and good for faster entry, gesture based computing.
- sleepychu 11y agoLearning sign language as an adult and using it a lot can give you really terrible RSI.
- melling 11y agoSomeone could devise a more subtle set of movements for text entry. Google's Soli would recognize small movements: http://www.youtube.com/watch?v=0QNiZfSsPc0 http://www.youtube.com/watch?v=0QNiZfSsPc0 There are others: https://github.com/melling/ErgonomicNotes#gesture-computing https://github.com/melling/ErgonomicNotes#gesture-computing
- drewbug01 11y agoIf you're getting RSI from sign language you're either an interpreter or over-producing. In the case of the former, you're using the language differently than a typical user. Most people don't produce language in the same amounts or in the same ways that an interpreter will (that is; speaking/signing isn't most people's job). There are steps you can take to reduce RSI as an interpreter, including proper rest while working and supportive braces at night. For the latter, it's possible to over-produce signs - basically "signing too hard." That can be painful, but it's correctable.
- jallmann 11y ago> Most people don't produce language in the same amounts that an interpreter will It isn't really because of how much they sign (deaf people don't typically get RSI from sign language), but rather because of how they sign it. The signing is more precise and formal, the hands/arms are elevated more, and generally less "colloquial" than you would see in day-to-day sign language.
- fareesh 11y agoThis is a really novel approach, but IMHO a more well rounded solution is the one being developed at Microsoft Research using Kinect. https://www.youtube.com/watch?v=HnkQyUo3134 https://www.youtube.com/watch?v=HnkQyUo3134 Nonetheless, great work by these students. I think their work is likely to have more applications in other areas where mimicking the movement of hands and fingers is an essential component.
- macu 11y agoI would love a new text entry method using gloves or finger/toe trackers. For recording thoughts while on the go.
- scholia 11y agoHave you tried a chorded keyboard? https://en.wikipedia.org/wiki/Chorded_keyboard https://en.wikipedia.org/wiki/Chorded_keyboard
- gloves 11y agoI don't translate sign language into text and speech!
- Zuider 11y agoDon't know why you got downvoted. Your comment is both admirably terse, and completely valid! SL is an entirely distinct language. People who are fluent in multiple languages switch from linguistic context to another when they speak different languages. They don't translate from their native language. So it makes sense that you don't do this either.
- breezest 11y agoI heard a similar work done by the students in Chinese University university of Hong Kong. But they make their own gloves, which cost about $75. https://translate.google.com/translate?sl=zh-CN&tl=en&js=y&prev=_t&hl=zh-TW&ie=UTF-8&u=https%3A%2F%2Fwww.cse.cuhk.edu.hk%2Fv7%2Fb5%2Fabout%2Fachieve%2F84.html&edit-text= https://translate.google.com/translate?sl=zh-CN&tl=en&js=y&p...
- xfour 11y agoCongo was way ahead of its time. Amy good Gorilla! http://m.youtube.com/watch?v=KXlSFKLuzTA http://m.youtube.com/watch?v=KXlSFKLuzTA
- c3534l 11y agoThere hasn't been a better movie with laser-mounted gorillas since.
- AdamSC1 11y agoIt's a great project, but to nail this they are going to need more involvement from the deaf community as they've missed some key points: 1)ASL isn't just about hand movements, there is a ton of grammar that comes from facial expression. It's not just like English tone, but can actually change a word. Like can become "don't like" from facial expression. 2) Context is very key, there are a lot of words with multiple meanings that matter based on the sentence. It could be figured out from a really smart system but wouldn't be easy. 3) Lastly, ASL isn't English. This is the biggest mistake people make. You aren't simply putting words into symbols. ASL has it's own grammar, syntax and even slang. I remember spending 15 minutes trying to figure out what my ASL teacher was saying when he said he "ate champ burger" and it turns out he meant it was the best burger he ever had. To 'translate' ASL you need to move from symbols to words and then add in the context. Ultimately there is a reason that professionals in this industry are called "sign language interpreters" and not "sign language translators" ASL is really hard to put into spoken or written language. It's also worth noting at really high level events an ASL interpreter will translate spoken word into sign and then a "Certified Deaf Interpreter" will re-sign it because the ASL of someone who is has communicated in sign their entire life has far more facial expression and meaning than even professional interpreters who are hearing. I think it's great to see advancements in technology trying to assist the deaf community, but I think it's important to get early involvement rather than assume we know what people need. I hope they keep up the great work - it's on the right path.
- gugagore 11y agoActually, the distinction (that I've seen) between "translator" and "interpreter" is whether the "transformation" is happening in real-time across spoken/sign language, or whether it's written language. Translators take text in one language and write it in another, like translating a legal document. Interpreters consume speech/sign in one language and speak/sign it in another.
- stevenbedrick 11y agoA useful commentary from a linguist who studies ASL: http://katies.online/katiesblog/index.php/2016/04/27/yes-ive-seen-the-signing-gloves/ http://katies.online/katiesblog/index.php/2016/04/27/yes-ive...
- scholia 11y agoThere's a longer and more informative story at UW Today: http://www.washington.edu/news/2016/04/12/uw-undergraduate-team-wins-10000-lemelson-mit-student-prize-for-gloves-that-translate-sign-language/ http://www.washington.edu/news/2016/04/12/uw-undergraduate-t...
- codingdave 11y agoWith all the critiques in the comments so far, it is worth noting that this article simply says they won a contest for being promising inventors, not that the invention doesn't have flaws.
- gugagore 11y agoDid you watch the corresponding video