5 ms·

Microsoft Research turns Kinect into sign language reader

65 points by tellarin 13y ago

jcater 13y ago
I wonder how it would distinguish between the many dialects in use in the US. Granted, a lot of the adult population does use ASL [1], but lots of times children grow up using one of the Signed English[2] variants, which vary from state to state. [1] http://en.wikipedia.org/wiki/American_Sign_Language http://en.wikipedia.org/wiki/American_Sign_Language [2] http://en.wikipedia.org/wiki/Signing_Exact_English http://en.wikipedia.org/wiki/Signing_Exact_English [Edit: links]
- stephengillie 13y ago
  Regional Language Packs?
lucb1e 13y ago
Well that's gotta be the first useful application of the hardware that I've heard of! I've tried Kinect before and it might be just me, but I find it to be horrible for gaming. Give me physical keys and I'm all happy. But of course that doesn't appeal to the bigger audience of casual gamers. Nintendo got that right with their, now dust-covered, Wiis.
- kvb 13y ago
  There have been a few other interesting applications, such as use by surgeons where physically using a computer would be unhygienic (e.g. see http://www.zdnet.com/blog/health/xbox-kinect-helps-surgeons-in-the-operating-room/277 http://www.zdnet.com/blog/health/xbox-kinect-helps-surgeons-...).
- FrojoS 13y ago
  Kinect is used a lot in rehabilitation and assistance of the impaired. Research conferences are full of Kinect applications and commercial applications are about to come on the market. Just today I found out about this great company which allows to control the mouse with head movements and facial expressions. http://kinesicmouse.xcessity.at/ http://kinesicmouse.xcessity.at/ PS: In my opinion the uselessness of the current Kinect for gaming is mostly due to the latency (up to 1s!) and to a lesser extend to the accuracy of the tracking.
- lloeki 13y ago
  None of that is gesture based but: - Forza head-tracking is really immersive and extends your screen. - Ally control by voice is great too because it's a lot more expressive and parallel than buttons (and it frees up buttons).
lukifer 13y ago
I'd be very curious to learn about the speed difference between a competent typist and a fluent signer, especially if visual cues can be used to accelerate the latter (ie, an equivalent of auto-complete). When writing prose (as opposed to code), I could actually see signing being very speed-competitive, and possibly a more natural way to communicate text than a keyboard. (Other advantages might be use with Glass, or the ability to pace around a room while "typing".) Maybe someday, kids will learn signing instead of keyboarding.
- seiji 13y ago
  When writing prose (as opposed to code), I could actually see signing being very speed-competitive How so? Signing isn't as "complete" as written or spoken language. You would say/write "The dog ran in the park yesterday" but the signing would be more like "YESTERDAY, DOG RUN [THROUGH] PARK." You're not going to get David Mitchell out of ASL, but then again you can't get motive music videos in Dejima either. They each have their place and meaning, but they aren't quite interchangeable.
  - lukifer 13y ago
    Right, as a non-signer it didn't even occur that syntax would be different, implied, or occasionally non-existent. In theory AI could try to auto-generate, but yeah, maybe not as rosy as I hoped.
- rasur 13y ago
  My son is 2 and recently found to be deaf. One usage case I imagined (while we are going through the process of learning sign-language and learning about the forthcoming Cochlea Implant he will receive) is exactly this, so it's gratifying to see some development in that sphere. Sign-language is different to English in the same way German is different to English.. you can translate, but it's not quite the same. Regardless, I did wonder - given how cretinous most Human / Computer Interfaces still are - if learning signing would be a better way of interacting with a machine. It would certainly have the side-benefit that users could talk to deaf people more easily (and possibly would help reduce the stigma that being deaf seems to still have, even today).
mabbo 13y ago
Having studied ASL at University (almost got a minor in it) I have a bit of a hard time believing this is fully possible at all. Signed languages aren't simply a series of words where each word has a given sign. It's a very complex and nuances language, and the regional dialects alone would make it tough to really decide what sign means what noun or verb.
- ericd 13y ago
  Probabilistic graphical models like Markov networks can yield surprisingly complex behaviors, if they're complex enough. I don't have much trouble believing that it's possible.
  - Argorak 13y ago
    Considering that sign languages regularly invent signs, e.g. for names, I doubt that. The interpreter might find out that there is a name at that place, but not know what the written equivalent of that name is.
    - seiji 13y ago
      Nobody is arguing complete all-knowing recognition of human movement. It's just recognition of a sign for "table" being table and a sign for "hat" being hat, etc.
      - ericd 13y ago
        And PGMs make it so you can use context to disambiguate things, so it can go beyond 1:1 sign-word recognition. Not sure if the researchers here are doing that yet, though.
      - mabbo 13y ago
        Except that is the very problem: drive 100 miles, and the signs for table and hat are different. Change the conversation, and the signs may be different. I can put an arm low and flat, and it means a general flat surface. It doesn't mean table until I mime setting a cup down on it, for example. Like I said, complex and nuanced. You have some incorrect assumptions in how signed languages work.
volandovengo 13y ago
It is a shame that things like this aren't made into shipping products. Maybe at some point they will integrate research into the product groups so that we can still really cool (self driving car cool) projects coming out of Microsoft
tn13 13y ago
During one coruse project we had used Kinect as a Semaphore (the flag signalling) reader.
- steve19 13y ago
  I got to play with a prototype simulator at a Navy base where a Kinect was used to allow cross training of helicopter pilots and those signaling guys who direct the helicopter from the deck of a ship. The signaling guy stands in front of a large wrap-around projector screen with a Kinect to capture his arm movements, which were sent over the network to a central server which ran the simulation. The pilot in the heli cockpit simulator would see the guy on the ship. The pilot and signal guy could be training on different sides of the country (or world).
spellboots 13y ago
Another win for the NSA against the menace of deaf terrorists!

65 points by tellarin 13y ago

I wonder how it would distinguish between the many dialects in use in the US. Granted, a lot of the adult population does use ASL [1], but lots of times children grow up using one of the Signed English[2] variants, which vary from state to state. [1] http://en.wikipedia.org/wiki/American_Sign_Language http://en.wikipedia.org/wiki/American_Sign_Language [2] http://en.wikipedia.org/wiki/Signing_Exact_English http://en.wikipedia.org/wiki/Signing_Exact_English [Edit: links]

stephengillie 13y ago
Regional Language Packs?

lucb1e 13y ago

Well that's gotta be the first useful application of the hardware that I've heard of! I've tried Kinect before and it might be just me, but I find it to be horrible for gaming. Give me physical keys and I'm all happy. But of course that doesn't appeal to the bigger audience of casual gamers. Nintendo got that right with their, now dust-covered, Wiis.

kvb 13y ago
There have been a few other interesting applications, such as use by surgeons where physically using a computer would be unhygienic (e.g. see http://www.zdnet.com/blog/health/xbox-kinect-helps-surgeons-in-the-operating-room/277 http://www.zdnet.com/blog/health/xbox-kinect-helps-surgeons-...).
FrojoS 13y ago
Kinect is used a lot in rehabilitation and assistance of the impaired. Research conferences are full of Kinect applications and commercial applications are about to come on the market. Just today I found out about this great company which allows to control the mouse with head movements and facial expressions. http://kinesicmouse.xcessity.at/ http://kinesicmouse.xcessity.at/ PS: In my opinion the uselessness of the current Kinect for gaming is mostly due to the latency (up to 1s!) and to a lesser extend to the accuracy of the tracking.
lloeki 13y ago
None of that is gesture based but: - Forza head-tracking is really immersive and extends your screen. - Ally control by voice is great too because it's a lot more expressive and parallel than buttons (and it frees up buttons).

lukifer 13y ago

I'd be very curious to learn about the speed difference between a competent typist and a fluent signer, especially if visual cues can be used to accelerate the latter (ie, an equivalent of auto-complete). When writing prose (as opposed to code), I could actually see signing being very speed-competitive, and possibly a more natural way to communicate text than a keyboard. (Other advantages might be use with Glass, or the ability to pace around a room while "typing".) Maybe someday, kids will learn signing instead of keyboarding.

seiji 13y ago
When writing prose (as opposed to code), I could actually see signing being very speed-competitive How so? Signing isn't as "complete" as written or spoken language. You would say/write "The dog ran in the park yesterday" but the signing would be more like "YESTERDAY, DOG RUN [THROUGH] PARK." You're not going to get David Mitchell out of ASL, but then again you can't get motive music videos in Dejima either. They each have their place and meaning, but they aren't quite interchangeable.
- lukifer 13y ago
  Right, as a non-signer it didn't even occur that syntax would be different, implied, or occasionally non-existent. In theory AI could try to auto-generate, but yeah, maybe not as rosy as I hoped.
rasur 13y ago
My son is 2 and recently found to be deaf. One usage case I imagined (while we are going through the process of learning sign-language and learning about the forthcoming Cochlea Implant he will receive) is exactly this, so it's gratifying to see some development in that sphere. Sign-language is different to English in the same way German is different to English.. you can translate, but it's not quite the same. Regardless, I did wonder - given how cretinous most Human / Computer Interfaces still are - if learning signing would be a better way of interacting with a machine. It would certainly have the side-benefit that users could talk to deaf people more easily (and possibly would help reduce the stigma that being deaf seems to still have, even today).

mabbo 13y ago

Having studied ASL at University (almost got a minor in it) I have a bit of a hard time believing this is fully possible at all. Signed languages aren't simply a series of words where each word has a given sign. It's a very complex and nuances language, and the regional dialects alone would make it tough to really decide what sign means what noun or verb.

ericd 13y ago
Probabilistic graphical models like Markov networks can yield surprisingly complex behaviors, if they're complex enough. I don't have much trouble believing that it's possible.
- Argorak 13y ago
  Considering that sign languages regularly invent signs, e.g. for names, I doubt that. The interpreter might find out that there is a name at that place, but not know what the written equivalent of that name is.
  - seiji 13y ago
    Nobody is arguing complete all-knowing recognition of human movement. It's just recognition of a sign for "table" being table and a sign for "hat" being hat, etc.
    - ericd 13y ago
      And PGMs make it so you can use context to disambiguate things, so it can go beyond 1:1 sign-word recognition. Not sure if the researchers here are doing that yet, though.
    - mabbo 13y ago
      Except that is the very problem: drive 100 miles, and the signs for table and hat are different. Change the conversation, and the signs may be different. I can put an arm low and flat, and it means a general flat surface. It doesn't mean table until I mime setting a cup down on it, for example. Like I said, complex and nuanced. You have some incorrect assumptions in how signed languages work.

volandovengo 13y ago

It is a shame that things like this aren't made into shipping products. Maybe at some point they will integrate research into the product groups so that we can still really cool (self driving car cool) projects coming out of Microsoft

tn13 13y ago

During one coruse project we had used Kinect as a Semaphore (the flag signalling) reader.

steve19 13y ago
I got to play with a prototype simulator at a Navy base where a Kinect was used to allow cross training of helicopter pilots and those signaling guys who direct the helicopter from the deck of a ship. The signaling guy stands in front of a large wrap-around projector screen with a Kinect to capture his arm movements, which were sent over the network to a central server which ran the simulation. The pilot in the heli cockpit simulator would see the guy on the ship. The pilot and signal guy could be training on different sides of the country (or world).

spellboots 13y ago

Another win for the NSA against the menace of deaf terrorists!