2 ms·

I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movements. If this mapping differed from a r

I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movements. If this mapping differed from a regular person this system would not be that useful on the interpreting speech side. I think we've had pretty good gesture recognition for a while, on the other hand. I bet it's possible to decode individual signs right now but sign language also has a different grammar from typical spoken English and a lot of meaning is context based so it might be tricky in that way, more of a translation problem.

padolsey 3y ago
Oh yeh definitely. I meant more specifically: might it be possible to capture the electrical signals (much like this current system) from the parts of the motor cortex that create the series of muscle movements that form a 'sign', and then creating a 2d projection/display of these muscle movements and then ... downstream, a gesture recognition solution as u mention (a big downstream challenge). It sounds like a lot. Was just a thought experiment of how such spinal blocks/paralysis would affect deaf people and how they'd be able to continue to communicate with their deaf partners. Definitely niche but nonetheless interesting, and I think possible using the same general approach as the OP article. But yeh to then translate gestures to speech is a distinct and incredibly challenging problem on its own as you allude to. Perhaps in the future they can tap into signing/speaking-translators' brains and have AI learn those mappings in a fuzzy way.
retrac 3y ago
> I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movement In fluent sign language, there is something analogous to phonemes. In linguistics these days, they're just called phonemes, and considered equivalent to spoken language phonemes. They're a fixed class of shapes and locations. They combine in certain ways that make up morphemes, which then make up words. It does work very similarly, perhaps identically, to spoken language. The distribution of handshapes and they way they interact resembles spoken language. For example, it's somewhat hard to say "strengths" and people often produce a slurred "strengfs". The way it slurs together is rather predictable. It's very hard to say "klftggt", and so it just doesn't occur in natural language. Same with sign languages and hard-to-sign combinations. Phonemes have an exact realization, but they also exist relative to each other, the distance and direction between them is important. This is probably part of why an American can fairly easily understand New Zealand English, despite nearly all of the vowels being different. Another analogy: in tonal languages, if there's a low flat tone, then 3 rising tones, then a low flat tone, that final low tone may be quite a bit higher than the first low tone -- but it will be interpreted as a low tone, as it is judged relative to the previous rising tone. Vowel qualities besides tone work the same way. And so do hand gestures. There is a lot of variation by dialect/region/community in sign languages. More than in a language like English. This makes it more complicated, but it shouldn't be insurmountable. And of course, not all deaf people speak sign languages as their native language. They would struggle just as people who learn any other language later in life do.
coldtea 3y ago
>I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movements. If this mapping differed from a regular person this system would not be that useful on the interpreting speech side. ...unless it could be trained on the individual?

padolsey 3y ago

Oh yeh definitely. I meant more specifically: might it be possible to capture the electrical signals (much like this current system) from the parts of the motor cortex that create the series of muscle movements that form a 'sign', and then creating a 2d projection/display of these muscle movements and then ... downstream, a gesture recognition solution as u mention (a big downstream challenge). It sounds like a lot. Was just a thought experiment of how such spinal blocks/paralysis would affect deaf people and how they'd be able to continue to communicate with their deaf partners. Definitely niche but nonetheless interesting, and I think possible using the same general approach as the OP article. But yeh to then translate gestures to speech is a distinct and incredibly challenging problem on its own as you allude to. Perhaps in the future they can tap into signing/speaking-translators' brains and have AI learn those mappings in a fuzzy way.

retrac 3y ago

> I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movement In fluent sign language, there is something analogous to phonemes. In linguistics these days, they're just called phonemes, and considered equivalent to spoken language phonemes. They're a fixed class of shapes and locations. They combine in certain ways that make up morphemes, which then make up words. It does work very similarly, perhaps identically, to spoken language. The distribution of handshapes and they way they interact resembles spoken language. For example, it's somewhat hard to say "strengths" and people often produce a slurred "strengfs". The way it slurs together is rather predictable. It's very hard to say "klftggt", and so it just doesn't occur in natural language. Same with sign languages and hard-to-sign combinations. Phonemes have an exact realization, but they also exist relative to each other, the distance and direction between them is important. This is probably part of why an American can fairly easily understand New Zealand English, despite nearly all of the vowels being different. Another analogy: in tonal languages, if there's a low flat tone, then 3 rising tones, then a low flat tone, that final low tone may be quite a bit higher than the first low tone -- but it will be interpreted as a low tone, as it is judged relative to the previous rising tone. Vowel qualities besides tone work the same way. And so do hand gestures. There is a lot of variation by dialect/region/community in sign languages. More than in a language like English. This makes it more complicated, but it shouldn't be insurmountable. And of course, not all deaf people speak sign languages as their native language. They would struggle just as people who learn any other language later in life do.

coldtea 3y ago

>I think not all deaf people have the same mapping of words and their precise phoneme to the typical expected muscle movements. If this mapping differed from a regular person this system would not be that useful on the interpreting speech side. ...unless it could be trained on the individual?