5 ms·

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly cl

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translation and we are good to go. But where can we find this data? Luckily news broadcasters often include special news segments for the hearing-impaired. You need _way_ more than just 5000 hours of video. People who are deaf of hard of hearing, in my experience, dislike the interpreters in news broadcasts. It's very difficult, as an interpreter, to provide _worthwhile_ translations of what is being spoken _as_ it is being spoken. It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. The other issue is most interpreters are hearing and so use the language slightly differently from actual deaf persons, and training on this on news topics will make it very weak when it comes to understanding and interpreting anything outside of this context. ASL has "dialects" and "slang." Hearing people always presume this will be simple. They should really just take an ASL class and worth with deaf and hearing impaired people first.

bluGill 2y ago
Lifeprint.org has plenty of free asl courses taught by a deaf person. Highly recommended for everyone but as with any language it takes a lot of study to be useful.
- jazzyjackson 2y ago
  .org landed on a squatting page, I suppose you mean https://lifeprint.com/asl101/lessons/lessons.htm https://lifeprint.com/asl101/lessons/lessons.htm
  - wonger_ 2y ago
    Just spent 5 minutes following along with the first video. Very clear and friendly instructor
voidingw 2y ago
The blog post references translating between English and Indian Sign Language (ISL). I interpreted that to mean translating between spoken English and ISL, not ASL and ISL. Regardless, I’m curious how (dis)similar ISL is to ASL.
- matroid 2y ago
  That is correct. We want to translate between English and ISL. English, because it is by and large the language of the Web and I think we should try to connect ISL to it rather than Indian Languages. From my understanding, they are quite dissimilar. A person who knows ISL will not understand ASL, for example.
  - thaumasiotes 2y ago
    > A person who knows ISL will not understand ASL, for example. That isn't the best example; it would still be true of language pairs that were very similar to each other.
al_borland 2y ago
I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL. During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID, because she refused to wear a mask while working, because coving up the face would make it more difficult to convey the message. I had to respect the dedication.
- refulgentis 2y ago
  I got a insanely dumb question, that probably has an obvious answer: why is it so critically important to have ASL? It seems to me you could skip having someone to the side adding the layers of emoting imperfectly and just watch the anchor and captions? Note, I'm definitely the one missing something. When first met with a contradiction, check your premises, is my motto, not lecture on why they're wrong.
  - amdsn 2y ago
    Think of it more like "why is it important to have Spanish as an option for captions in an area that knows it has a large audience of L1 Spanish speakers?". English/<insert dominant local language> is pretty much a foreign language to many signers. Their native language is ASL or whatever other sign language they know, and these languages aren't just 1:1 mappings of words in the local dominant language to hand signals. They have many dimensions of expression for encoding meaning like facial expression/body motions, speed of the sign, amount of times they repeat the sign that have grammatical meaning that spoken languages express with inflection (noun cases, different verb forms) or additional words. For example, where an English speaker would use words like "very" or "extremely" or choose adjectives with more intense connotations an ASL signer would repeat or exaggerate the sign they want to emphasize (often by signing it more quickly, but it frequently involves intensifying multiple parts of the sign like the entire motion or the facial expression as well).
    - 2y ago
deleted 2y ago
[deleted]
matroid 2y ago
Thanks for the feedback. You raise great points and this was the reason why we wrote this post, so that we can hear from people where the actual problem lies. On a related note, this sort of explains why our model is struggling to fit on 500 hours of our current dataset (even on the training set). Even so, the current state of automatic translation for Indian Sign Language is that, in-the-wild, even individual words cannot be detected very well. We hope that what we are building might at least improve the state-of-the-art there. > It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. Can you elaborate a bit more on this. Do you think if we make a system for bad/broken transliteration and funnel it through ChatGPT, it might give meaningful results? That is ChatGPT might be able to correct for errors as it is a strong language model.
- wizzwizz4 2y ago
  I think you think it's a magic box. There's not actually such thing as a "strong language model", not in the way you're using the concept. > We hope that what we are building might at least improve the state-of-the-art there. Do you have any theoretical arguments for how and why it would improve it? If not, my concern is that you're just sucking the air out of the room. (Research into "throw a large language model at the problem" doesn't tend to produce any insight that could be used by other approaches, and doesn't tend to work, but it does funnel a lot of grant funding into cloud providers' pockets.)
  - canjobear 2y ago
    “Throw an LM at it” is the only approach that has ever produced human level machine translation. For theory on how a strong target-language-side LM can improve translation, even in the extreme scenario where no parallel “texts” are available, https://proceedings.neurips.cc/paper_files/paper/2023/file/7571c9d44179c7988178593c5b62a9b6-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2023/file/7...
    - wizzwizz4 2y ago
      You're mixing up cause and effect. The transformer architecture was invented for machine translation – and it's pretty good at it! (Very far from human-level, but still mostly comprehensible, and a significant improvement over the state-of-the-art at time of first publication.) But we shouldn't treat this as anything more than "special-purpose ML architecture achieves decent results". The GPT architecture, using transformers to do iterated predictive text, is a modern version of the Markov bot. It's truly awful at translation, when "prompted" to do so. (Perhaps surprisingly so, until you step back, look at the training data, and look at the information flow: the conditional probability of the next token isn't mostly coming from the source text.) I haven't read that paper yet, but it looks interesting. From the abstract, it looks like one of those perfectly-valid papers that laypeople think is making a stronger claim than it is. This paragraph supports that: > Note that these models are not intended to accurately capture natural language. Rather, they illustrate how our theory can be used to study the effect of language similarity and complexity on data requirements for UMT.
kobalsky 2y ago
> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?
- jallmann 2y ago
  Many deaf people do prefer sign language as an accessibility option over reading captions, even if the interpreting can be hit-or-miss.
- matroid 2y ago
  Also, in India, many hearing-impaired people know only ISL.
  - lolinder 2y ago
    This is also true in the US. People pick up enough written English to get by most of the time, but it's often quite broken and clearly a second language. I know hearing impaired people native to the US with substantially worse English than the average European.
  - egberts1 2y ago
    Just so you know, "hearing-impaired" implies that a person has a flaw whether a person is born with it (it is natural to them) or impacted in later life (hearing-challenged). Most non-offensive way to refer to a group of people without perfect hearing is "hard-of-hearing or deaf".
- bluGill 2y ago
  Sign languages is nothing like spoken language and so reading a spoken language is hard.
WesternWind 2y ago
Just to note this is for ISL, Indian Sign Language, not ASL, American Sign Language.
School-Cotton 2y ago
> American Sign Language is not English I'm not sure I understand your point. Chinese is also not English but machine translation of Chinese to English can be done. You're right that laypeople often assume, wrongly, that a given country's sign language is an encoding of the local spoken language. In reality it's usually a totally different language in its own right. But that shouldn't mean that translation is fundamentally impossible.
- lolinder 2y ago
  They didn't say it's fundamentally impossible, they said you need way more than 5000 hours to do it and that you'd need better quality sources than news interpreters.

bluGill 2y ago

Lifeprint.org has plenty of free asl courses taught by a deaf person. Highly recommended for everyone but as with any language it takes a lot of study to be useful.

jazzyjackson 2y ago
.org landed on a squatting page, I suppose you mean https://lifeprint.com/asl101/lessons/lessons.htm https://lifeprint.com/asl101/lessons/lessons.htm
- wonger_ 2y ago
  Just spent 5 minutes following along with the first video. Very clear and friendly instructor

voidingw 2y ago

The blog post references translating between English and Indian Sign Language (ISL). I interpreted that to mean translating between spoken English and ISL, not ASL and ISL. Regardless, I’m curious how (dis)similar ISL is to ASL.

matroid 2y ago
That is correct. We want to translate between English and ISL. English, because it is by and large the language of the Web and I think we should try to connect ISL to it rather than Indian Languages. From my understanding, they are quite dissimilar. A person who knows ISL will not understand ASL, for example.
- thaumasiotes 2y ago
  > A person who knows ISL will not understand ASL, for example. That isn't the best example; it would still be true of language pairs that were very similar to each other.

al_borland 2y ago

I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL. During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID, because she refused to wear a mask while working, because coving up the face would make it more difficult to convey the message. I had to respect the dedication.

refulgentis 2y ago
I got a insanely dumb question, that probably has an obvious answer: why is it so critically important to have ASL? It seems to me you could skip having someone to the side adding the layers of emoting imperfectly and just watch the anchor and captions? Note, I'm definitely the one missing something. When first met with a contradiction, check your premises, is my motto, not lecture on why they're wrong.
- amdsn 2y ago
  Think of it more like "why is it important to have Spanish as an option for captions in an area that knows it has a large audience of L1 Spanish speakers?". English/<insert dominant local language> is pretty much a foreign language to many signers. Their native language is ASL or whatever other sign language they know, and these languages aren't just 1:1 mappings of words in the local dominant language to hand signals. They have many dimensions of expression for encoding meaning like facial expression/body motions, speed of the sign, amount of times they repeat the sign that have grammatical meaning that spoken languages express with inflection (noun cases, different verb forms) or additional words. For example, where an English speaker would use words like "very" or "extremely" or choose adjectives with more intense connotations an ASL signer would repeat or exaggerate the sign they want to emphasize (often by signing it more quickly, but it frequently involves intensifying multiple parts of the sign like the entire motion or the facial expression as well).
  - 2y ago

deleted 2y ago

[deleted]

matroid 2y ago

Thanks for the feedback. You raise great points and this was the reason why we wrote this post, so that we can hear from people where the actual problem lies. On a related note, this sort of explains why our model is struggling to fit on 500 hours of our current dataset (even on the training set). Even so, the current state of automatic translation for Indian Sign Language is that, in-the-wild, even individual words cannot be detected very well. We hope that what we are building might at least improve the state-of-the-art there. > It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. Can you elaborate a bit more on this. Do you think if we make a system for bad/broken transliteration and funnel it through ChatGPT, it might give meaningful results? That is ChatGPT might be able to correct for errors as it is a strong language model.

wizzwizz4 2y ago
I think you think it's a magic box. There's not actually such thing as a "strong language model", not in the way you're using the concept. > We hope that what we are building might at least improve the state-of-the-art there. Do you have any theoretical arguments for how and why it would improve it? If not, my concern is that you're just sucking the air out of the room. (Research into "throw a large language model at the problem" doesn't tend to produce any insight that could be used by other approaches, and doesn't tend to work, but it does funnel a lot of grant funding into cloud providers' pockets.)
- canjobear 2y ago
  “Throw an LM at it” is the only approach that has ever produced human level machine translation. For theory on how a strong target-language-side LM can improve translation, even in the extreme scenario where no parallel “texts” are available, https://proceedings.neurips.cc/paper_files/paper/2023/file/7571c9d44179c7988178593c5b62a9b6-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2023/file/7...
  - wizzwizz4 2y ago
    You're mixing up cause and effect. The transformer architecture was invented for machine translation – and it's pretty good at it! (Very far from human-level, but still mostly comprehensible, and a significant improvement over the state-of-the-art at time of first publication.) But we shouldn't treat this as anything more than "special-purpose ML architecture achieves decent results". The GPT architecture, using transformers to do iterated predictive text, is a modern version of the Markov bot. It's truly awful at translation, when "prompted" to do so. (Perhaps surprisingly so, until you step back, look at the training data, and look at the information flow: the conditional probability of the next token isn't mostly coming from the source text.) I haven't read that paper yet, but it looks interesting. From the abstract, it looks like one of those perfectly-valid papers that laypeople think is making a stronger claim than it is. This paragraph supports that: > Note that these models are not intended to accurately capture natural language. Rather, they illustrate how our theory can be used to study the effect of language similarity and complexity on data requirements for UMT.

kobalsky 2y ago

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

jallmann 2y ago
Many deaf people do prefer sign language as an accessibility option over reading captions, even if the interpreting can be hit-or-miss.
matroid 2y ago
Also, in India, many hearing-impaired people know only ISL.
- lolinder 2y ago
  This is also true in the US. People pick up enough written English to get by most of the time, but it's often quite broken and clearly a second language. I know hearing impaired people native to the US with substantially worse English than the average European.
- egberts1 2y ago
  Just so you know, "hearing-impaired" implies that a person has a flaw whether a person is born with it (it is natural to them) or impacted in later life (hearing-challenged). Most non-offensive way to refer to a group of people without perfect hearing is "hard-of-hearing or deaf".
bluGill 2y ago
Sign languages is nothing like spoken language and so reading a spoken language is hard.

WesternWind 2y ago

Just to note this is for ISL, Indian Sign Language, not ASL, American Sign Language.

School-Cotton 2y ago

> American Sign Language is not English I'm not sure I understand your point. Chinese is also not English but machine translation of Chinese to English can be done. You're right that laypeople often assume, wrongly, that a given country's sign language is an encoding of the local spoken language. In reality it's usually a totally different language in its own right. But that shouldn't mean that translation is fundamentally impossible.

lolinder 2y ago
They didn't say it's fundamentally impossible, they said you need way more than 5000 hours to do it and that you'd need better quality sources than news interpreters.