3 ms·

The hackathon: "let's make a machine that interprets sign language" trope is a personal bugbear of mine. Being able to read out finger spelling is kinda useles

The hackathon: "let's make a machine that interprets sign language" trope is a personal bugbear of mine. Being able to read out finger spelling is kinda useless when most of the relevant population can't spell (why bother with an abitrary phonetic encoding for words in a language you don't know) Newer systems can read entire signs, which is cooler, but still useless because they can't handle the grammar or context. Just some words in isolation. Don't waste time on this. This is not an accessability tool for the disabled, it is a "feel proud tool" for undergrads with a machine learning library. The real technological solution that would be helpful is still a few more major advancements away. It needs to handle grammar (ASL has radically different grammar than most spoken languages fyi). Going from a spoken language to a sign language is also significantly more useful. Go learn ASL (or your local sign language) and encourage others to learn. That effort will do more good in making the world accessible to those who depend on sign language to understand and be understood.

smeeth 4y ago
> The real technological solution that would be helpful is still a few more major advancements away. It needs to handle grammar (ASL has radically different grammar than most spoken languages fyi). Going from a spoken language to a sign language is also significantly more useful. FWIW I do ML research and these sound very possible with todays tech. I could probably hackathon this. This sounds more like a classic case of "engineer doesn't know what the users need" or "normie doesn't understand the actual life/needs of the disabled." It sounds like you should set up a hackathon! Or at least publish some information so that future hackathon participants don't fall into the common tropes.
- blamestross 4y ago
  By "different" I mean radically different. It's a different medium with different possibilities and challenges. I only know ASL other sign languages are less related to each other than spoken languages are (exception ASL derives strongly from French sign language) The root problem is that language concepts from spoken/written languages don't map 1-1 onto. Some examples: - "pronouns" (really "referring to anybody or anything who isn't here to point at) are spacial. Things get assigned a location in the conversation to be gestured towards to reference them. This requires a level of spacial recognition that doesn't exist yet outside of VR with a lot of hardware. (Extra challenge recognizing the location of things that ARE NOT ACTUALLY THERE) - descriptor/word order are very context sensitive (basically whatever order the signer thinks will refine things down for the watcher) It doesn't encode well to enable the alignment based translations. Encodings exist for teaching purposes, but they are not easy to use for more relaxed used (that's is what signing is for). Corpuses don't exist for that either. Modern translation techniques double down on having a large corpus and that basically all written languages have similar syntax structures even if the representations differ. Yes the "technology" exists in the same sense we know how to colonize the asteroids. We don't need a radical improvement of technology, just somebody to bother investing for decades without return. A hackathon isn't going to cross that void.
  - blamestross 4y ago
    Oh and I'm bad at it, so I forget, but facial expression and emoting are an entire grammar side-channel that we don't have machines to effectively emulate.
a2800276 4y ago
Does the library only handle finger spelling? I thought the pages were a little light on details and could use a video of a demo at the very least ... While I agree that most people naively approaching sign language vastly underestimate the challenge, I wouldn't consider it a waste of time, though. At least you learn a bit about machine learning and maybe with a bit of luck it raises awareness for the real challenges faced by sign language speakers.
- blamestross 4y ago
  Sure, but they rationalize and sell that "mildly useful educational content" with lies about how it will help people. I'm totally ok with somebody deciding to translate finger spelling as an educational exercise. Have at! Don't sell it as more than that, because it isn't.
  - a2800276 4y ago
    It will help people learn image recognition ;) I understand where you're coming from, though. A friend of mine is involved in Sign Language linguistics and everytime we meet she gripes about how much of her time is wasted by having to respond to AI companies who are supposedly inventing "revolutionary" technology to automate Sign Language translation.
iamwpj 4y ago
Spot on. If all the deaf community needed was rapid signing attached to presentations this issue would have been solved. Reliable subtitles have been around for years on TV with a short delay. ASL presentation is so much more than waving hands.

smeeth 4y ago

> The real technological solution that would be helpful is still a few more major advancements away. It needs to handle grammar (ASL has radically different grammar than most spoken languages fyi). Going from a spoken language to a sign language is also significantly more useful. FWIW I do ML research and these sound very possible with todays tech. I could probably hackathon this. This sounds more like a classic case of "engineer doesn't know what the users need" or "normie doesn't understand the actual life/needs of the disabled." It sounds like you should set up a hackathon! Or at least publish some information so that future hackathon participants don't fall into the common tropes.

blamestross 4y ago
By "different" I mean radically different. It's a different medium with different possibilities and challenges. I only know ASL other sign languages are less related to each other than spoken languages are (exception ASL derives strongly from French sign language) The root problem is that language concepts from spoken/written languages don't map 1-1 onto. Some examples: - "pronouns" (really "referring to anybody or anything who isn't here to point at) are spacial. Things get assigned a location in the conversation to be gestured towards to reference them. This requires a level of spacial recognition that doesn't exist yet outside of VR with a lot of hardware. (Extra challenge recognizing the location of things that ARE NOT ACTUALLY THERE) - descriptor/word order are very context sensitive (basically whatever order the signer thinks will refine things down for the watcher) It doesn't encode well to enable the alignment based translations. Encodings exist for teaching purposes, but they are not easy to use for more relaxed used (that's is what signing is for). Corpuses don't exist for that either. Modern translation techniques double down on having a large corpus and that basically all written languages have similar syntax structures even if the representations differ. Yes the "technology" exists in the same sense we know how to colonize the asteroids. We don't need a radical improvement of technology, just somebody to bother investing for decades without return. A hackathon isn't going to cross that void.
- blamestross 4y ago
  Oh and I'm bad at it, so I forget, but facial expression and emoting are an entire grammar side-channel that we don't have machines to effectively emulate.

a2800276 4y ago

Does the library only handle finger spelling? I thought the pages were a little light on details and could use a video of a demo at the very least ... While I agree that most people naively approaching sign language vastly underestimate the challenge, I wouldn't consider it a waste of time, though. At least you learn a bit about machine learning and maybe with a bit of luck it raises awareness for the real challenges faced by sign language speakers.

blamestross 4y ago
Sure, but they rationalize and sell that "mildly useful educational content" with lies about how it will help people. I'm totally ok with somebody deciding to translate finger spelling as an educational exercise. Have at! Don't sell it as more than that, because it isn't.
- a2800276 4y ago
  It will help people learn image recognition ;) I understand where you're coming from, though. A friend of mine is involved in Sign Language linguistics and everytime we meet she gripes about how much of her time is wasted by having to respond to AI companies who are supposedly inventing "revolutionary" technology to automate Sign Language translation.

iamwpj 4y ago

Spot on. If all the deaf community needed was rapid signing attached to presentations this issue would have been solved. Reliable subtitles have been around for years on TV with a short delay. ASL presentation is so much more than waving hands.