5 ms·

On a slightly related topic, India has a vast and rich set of languages and millions of people that speak/read/write these languages. With daily newspapers, tab

On a slightly related topic, India has a vast and rich set of languages and millions of people that speak/read/write these languages. With daily newspapers, tabloids, literature etc, I presume there is a (potential) huge corpus of rich data for each language too. And, I just read yesterday that India just surpassed US to become the second biggest mobile phone market. Then why is there not more of NLP work/products on Indian languages? There's a market for it, a huge market, isn't it?

true_religion 9y ago
Because as you said.. thee's a vast set of languages instead of a single one of 300 million people.
microcolonel 9y ago
> Then why is there not more of NLP work/products on Indian languages? There's a market for it, a huge market, isn't it? The market effects of linguistic diversity work both ways: having more languages increases the potential demand for translation tools, but the sheer number of long-tail languages also makes it hard to justify compiling a corpus and making a considerable effort to study each of these languages enough to build tools for them. Some language communities may almost completely lack the kind of highly literate people who would make this task surmountable, and those who could help, may already be working another more stable or higher paying job.
sandGorgon 9y ago
Here's something that was released fairly recently by my alma mater - IIT Bombay https://arxiv.org/abs/1710.02855 https://arxiv.org/abs/1710.02855
- dodigitty32 9y ago
  Is this anoop ? :P
axiom92 9y ago
> On a slightly related topic, India has a vast and rich set of languages and millions of people that speak/read/write these languages. My landlady needs to know this. She still thinks we all speak "Indian".
- afghanPower 9y ago
  Correct me if I'm wrong, but most Indians do understand "urdu", no?
  - axiom92 9y ago
    People with an understanding of Hindi should be able to understand everyday Urdu.
  - deanstag 9y ago
    I think you might be talking about Hindi. That is the most common language.
  - jfk13 9y ago
    That may be the case for many people across the "Hindi belt" of northern India. (Hindi and Urdu, at an everyday level, being fairly close to a case of "same spoken language, different scripts".) In the Dravidian-speaking south....not so much.
  - deleted 9y ago
    [deleted]
  - sumedh 9y ago
    Yes because most Indians can speak Hindi(Urdu is similar to Hindi). Hindi is the second or third language for many non native Hindi speakers.
    - wizardofmysore 9y ago
      Around 100+ million of 1.2 billion don't speak Hindi.
titanix2 9y ago
The current tools were created by generations of scientists and engineers; people that these linguistics communities lack. As some of them get educated in NLP and related fields, more resources and corpora will be build. But heck, even languages for which there is universities and labs working on it (plus EU funding) there is a huge gap. For example czech. As a side nodes, some existing researchers who have an interest in other languages start work on them even if it’s not their native language. I know researchers who did that with african languages: by working with students speaking them, there is transfer of knowledge and preliminary work done. Mentoring the first generation of native speaker scientists is the key imo.

true_religion 9y ago

Because as you said.. thee's a vast set of languages instead of a single one of 300 million people.

microcolonel 9y ago

> Then why is there not more of NLP work/products on Indian languages? There's a market for it, a huge market, isn't it? The market effects of linguistic diversity work both ways: having more languages increases the potential demand for translation tools, but the sheer number of long-tail languages also makes it hard to justify compiling a corpus and making a considerable effort to study each of these languages enough to build tools for them. Some language communities may almost completely lack the kind of highly literate people who would make this task surmountable, and those who could help, may already be working another more stable or higher paying job.

sandGorgon 9y ago

Here's something that was released fairly recently by my alma mater - IIT Bombay https://arxiv.org/abs/1710.02855 https://arxiv.org/abs/1710.02855

dodigitty32 9y ago
Is this anoop ? :P

axiom92 9y ago

> On a slightly related topic, India has a vast and rich set of languages and millions of people that speak/read/write these languages. My landlady needs to know this. She still thinks we all speak "Indian".

afghanPower 9y ago
Correct me if I'm wrong, but most Indians do understand "urdu", no?
- axiom92 9y ago
  People with an understanding of Hindi should be able to understand everyday Urdu.
- deanstag 9y ago
  I think you might be talking about Hindi. That is the most common language.
- jfk13 9y ago
  That may be the case for many people across the "Hindi belt" of northern India. (Hindi and Urdu, at an everyday level, being fairly close to a case of "same spoken language, different scripts".) In the Dravidian-speaking south....not so much.
- deleted 9y ago
  [deleted]
- sumedh 9y ago
  Yes because most Indians can speak Hindi(Urdu is similar to Hindi). Hindi is the second or third language for many non native Hindi speakers.
  - wizardofmysore 9y ago
    Around 100+ million of 1.2 billion don't speak Hindi.

titanix2 9y ago

The current tools were created by generations of scientists and engineers; people that these linguistics communities lack. As some of them get educated in NLP and related fields, more resources and corpora will be build. But heck, even languages for which there is universities and labs working on it (plus EU funding) there is a huge gap. For example czech. As a side nodes, some existing researchers who have an interest in other languages start work on them even if it’s not their native language. I know researchers who did that with african languages: by working with students speaking them, there is transfer of knowledge and preliminary work done. Mentoring the first generation of native speaker scientists is the key imo.