5 ms·

I can't read it, but what makes interpreting sign language so difficult? You can have a model that inputs two pictures of the person 5/60 seconds apart. If the

I can't read it, but what makes interpreting sign language so difficult? You can have a model that inputs two pictures of the person 5/60 seconds apart. If the change in movement is below a certain threshold(indicating a word is being displayed) then crop it around the hands and run it though a resnet trained to categorize images based off of words. Is there something I'm missing? If it's such a big deal then I may work on it myself.

schwartzworld 6y ago
American Sign Language grammar is subtle and complex.
hmry 6y ago
Sign languages don't only use hand shape, but also hand motion, body pose, facial expression, mouthing words, etc. Cropping still images around the hands won't do you much good at all.
- medstrom 6y ago
  Not to mention polysynthetic signs - I don't want to be the one to program that. An example sentence of mostly polysynthetic signs: "i was out driving, just enjoying the scenery, the nature and all the trees passing by me," (3 signs) "then suddenly" (just raise the eyebrows and look scared at some point in front of you, with your hands still on the imaginary steering wheel), "my car crashed into a wall (or other immobile object) (and the hood got all deformed)" - 1 sign. So that takes about 5 seconds to sign, but up to a minute to represent all the embedded information in English. But in practice, you'd skip a lot of the embedded information when translating to English, depending on what was actually important. So how do you make the machine know what was important? It depends on who you're translating for!
medstrom 6y ago
Essentially, vocal languages are one-dimensional, thus easy to solve. Filter out disturbances, background noise, and train the machine to adapt to dialects, and you're halfway home. Sign language is at least three-dimensional, it can embed more information. The ASL signs for "forget" and "because" differ only in how you mouth them - easy enough, but sometimes you even skip mouthing them because there is sufficient context. What's worse is the location of the hands can vary depending on how lazy the signer is feeling, so "forget" is supposed to be signed from the forehead but it can also be signed from some empty point in front of your shoulder, off a "phantom person"'s imaginary forehead. You rely on the listener to understand that you intend for there to be a phantom person there. Parsing sign language means you have to turn a 2D recording into a 3D model - even humans can have difficulty with this, parsing signs over Zoom - and you need to solve parsing facial expressions, reading lips and more. And you need the machine to be smart enough to detect the assignment of phantom persons to various locations around the signer. That's my impression as a deaf person anyway.
- kleiba 6y ago
  Essentially, vocal languages are one-dimensional, thus easy to solve. cough.
sohkamyung 6y ago
This paragraph from the article highlights some of the problems. > Research led by a team at Microsoft, a big computing firm, and published in 2019, estimated that a typical publicly available corpus of a spoken language consists of around a billion words from as many as 1,000 different speakers. An equivalent data-set in a sign language might have fewer than 100,000 signs from just ten people. Besides large numbers, a good corpus also needs variety. This means conversations between native signers of diverse backgrounds, dialects and levels of fluency. Because deaf people more often have physical disabilities than do those with unaffected hearing, representing those with restricted fluency of movement is important.
codyogden 6y ago
All sign languages use much more than handshape and position. And even outside of facial expression grammar (ie non-manual communication) there is a whole host of other complexities that make parsing sign language difficult. Take classifiers: there are eight morphological types that allow users to express complex ideas without the use of any type of vocabulary-based utterances. You can certainly sign the equivalent of, “Did you see the car parked close to the tree,” (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the point across, but it’s not efficient, and doesn’t really utilize sign language in a way that feels normal for users. But that same question can be posed in a more condensed format if you use a classifier, (#CAR 5:CL-“tree”/3:CL-“car parked really close” SEE?). In the second utterance, one use the lexicalized (fingerspelled “car”), then holds up a five hand with bent elbow in their non-dominant hand, use the three hand representing the car placing it reeeeeeally close (with grammatical, NMC emphasis) to the tree, and finally asking the receiver if they saw it. In that example alone, I’m using two types of classifiers: element and locative. Imagine a complex story using all eight classifier types with just a little vocabulary mixed in? I doubt AI/ML could interpret the context, characters, actions, events, descriptions, etc. accurately. Anyway, I hope this example shows there is a lot more complexity in computerized interpretation of signed languages well beyond gesture/vocabulary recognition. Note: My glossing may not be perfect. It's been a while, and I’m a bit rusty.
- codyogden 6y ago
  Correcting myself: The classifiers used in my example are "semantic and locative," not "element."
- whitten 6y ago
  Is this an ASL thing or some other system? What would be a good source to get the full list of the eight morphological types, and how they are used?
  - codyogden 6y ago
    I learned this in a Linguistics of ASL course as a part of my degree in ASL. My understanding is that classifiers exist in every natural signed language and isn’t exclusive to ASL. https://www.allenisd.org/cms/lib/TX01001197/Centricity/Domain/1900/Classifiers%20Notes.ppt https://www.allenisd.org/cms/lib/TX01001197/Centricity/Domai... The linked PowerPoint does a good job of explaining a brief overview of the different types of classifiers. (I actually referenced it to correct my own answer because I knew something felt off. lol)
PeterisP 6y ago
Fragmentation also is a big issue. Not only in the sense that there are separate very different sign languages (American Sign Language has little in common with British Sign Language, they are not mutually comprehensible), but even for each separate sign language you also get different 'dialects' in different areas/major cities, which are mutually comprehensible but have lexical variation depending on the speaker's location/community that needs to be properly represented in your datasets in order to be acknowledged and understood. In some sense it seems the equivalent of what the fragmentation was for spoken languages before the printing press and age of nationalism facilitated language standardization.

schwartzworld 6y ago

American Sign Language grammar is subtle and complex.

hmry 6y ago

Sign languages don't only use hand shape, but also hand motion, body pose, facial expression, mouthing words, etc. Cropping still images around the hands won't do you much good at all.

medstrom 6y ago
Not to mention polysynthetic signs - I don't want to be the one to program that. An example sentence of mostly polysynthetic signs: "i was out driving, just enjoying the scenery, the nature and all the trees passing by me," (3 signs) "then suddenly" (just raise the eyebrows and look scared at some point in front of you, with your hands still on the imaginary steering wheel), "my car crashed into a wall (or other immobile object) (and the hood got all deformed)" - 1 sign. So that takes about 5 seconds to sign, but up to a minute to represent all the embedded information in English. But in practice, you'd skip a lot of the embedded information when translating to English, depending on what was actually important. So how do you make the machine know what was important? It depends on who you're translating for!

medstrom 6y ago

Essentially, vocal languages are one-dimensional, thus easy to solve. Filter out disturbances, background noise, and train the machine to adapt to dialects, and you're halfway home. Sign language is at least three-dimensional, it can embed more information. The ASL signs for "forget" and "because" differ only in how you mouth them - easy enough, but sometimes you even skip mouthing them because there is sufficient context. What's worse is the location of the hands can vary depending on how lazy the signer is feeling, so "forget" is supposed to be signed from the forehead but it can also be signed from some empty point in front of your shoulder, off a "phantom person"'s imaginary forehead. You rely on the listener to understand that you intend for there to be a phantom person there. Parsing sign language means you have to turn a 2D recording into a 3D model - even humans can have difficulty with this, parsing signs over Zoom - and you need to solve parsing facial expressions, reading lips and more. And you need the machine to be smart enough to detect the assignment of phantom persons to various locations around the signer. That's my impression as a deaf person anyway.

kleiba 6y ago
Essentially, vocal languages are one-dimensional, thus easy to solve. cough.

sohkamyung 6y ago

This paragraph from the article highlights some of the problems. > Research led by a team at Microsoft, a big computing firm, and published in 2019, estimated that a typical publicly available corpus of a spoken language consists of around a billion words from as many as 1,000 different speakers. An equivalent data-set in a sign language might have fewer than 100,000 signs from just ten people. Besides large numbers, a good corpus also needs variety. This means conversations between native signers of diverse backgrounds, dialects and levels of fluency. Because deaf people more often have physical disabilities than do those with unaffected hearing, representing those with restricted fluency of movement is important.

codyogden 6y ago

All sign languages use much more than handshape and position. And even outside of facial expression grammar (ie non-manual communication) there is a whole host of other complexities that make parsing sign language difficult. Take classifiers: there are eight morphological types that allow users to express complex ideas without the use of any type of vocabulary-based utterances. You can certainly sign the equivalent of, “Did you see the car parked close to the tree,” (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the point across, but it’s not efficient, and doesn’t really utilize sign language in a way that feels normal for users. But that same question can be posed in a more condensed format if you use a classifier, (#CAR 5:CL-“tree”/3:CL-“car parked really close” SEE?). In the second utterance, one use the lexicalized (fingerspelled “car”), then holds up a five hand with bent elbow in their non-dominant hand, use the three hand representing the car placing it reeeeeeally close (with grammatical, NMC emphasis) to the tree, and finally asking the receiver if they saw it. In that example alone, I’m using two types of classifiers: element and locative. Imagine a complex story using all eight classifier types with just a little vocabulary mixed in? I doubt AI/ML could interpret the context, characters, actions, events, descriptions, etc. accurately. Anyway, I hope this example shows there is a lot more complexity in computerized interpretation of signed languages well beyond gesture/vocabulary recognition. Note: My glossing may not be perfect. It's been a while, and I’m a bit rusty.

codyogden 6y ago
Correcting myself: The classifiers used in my example are "semantic and locative," not "element."
whitten 6y ago
Is this an ASL thing or some other system? What would be a good source to get the full list of the eight morphological types, and how they are used?
- codyogden 6y ago
  I learned this in a Linguistics of ASL course as a part of my degree in ASL. My understanding is that classifiers exist in every natural signed language and isn’t exclusive to ASL. https://www.allenisd.org/cms/lib/TX01001197/Centricity/Domain/1900/Classifiers%20Notes.ppt https://www.allenisd.org/cms/lib/TX01001197/Centricity/Domai... The linked PowerPoint does a good job of explaining a brief overview of the different types of classifiers. (I actually referenced it to correct my own answer because I knew something felt off. lol)

PeterisP 6y ago

Fragmentation also is a big issue. Not only in the sense that there are separate very different sign languages (American Sign Language has little in common with British Sign Language, they are not mutually comprehensible), but even for each separate sign language you also get different 'dialects' in different areas/major cities, which are mutually comprehensible but have lexical variation depending on the speaker's location/community that needs to be properly represented in your datasets in order to be acknowledged and understood. In some sense it seems the equivalent of what the fragmentation was for spoken languages before the printing press and age of nationalism facilitated language standardization.