14 ms·

Seriously, I am trying my luck in NLP with some Indian languages like Marathi, Hindi, Gujarati and Tamil. The sheer number of dialects is driving me nuts. Even

Seriously, I am trying my luck in NLP with some Indian languages like Marathi, Hindi, Gujarati and Tamil. The sheer number of dialects is driving me nuts. Even dialects have very different sounding words or pronunciations which increases the complexity exponentially. But I am trying it only with an eye on its potential. NLP will simply act as the catalyst for technology adoption in the rural and semi-rural area. I'm thinking to study China's approach here since I heard even Chinese is incredibly diverse.

azurezyq 9y ago
China does have very diverse dialects. But Mandarin is taught in all schools in mainland China and we treat it as standard. Also it is the unified way people from different area communicate. For younger generation this is not a problem, back in my college life where people in the same class came from different areas, from Xinjiang to Canton to the northeast corner. We have no problem understanding each other though sometimes funny with unique words or accents. It seems that the problem is not as severe in China as in India.
- _emacsomancer_ 9y ago
  And what are called "dialects" in the Chinese context are just as much dialects as French, Italian, and Romanian are "dialects" of Romance.
  - jacobolus 9y ago
    These mostly get called “dialects” for political reasons: the Chinese government doesn’t want to acknowledge that they are separate mutually unintelligible languages. Media in Chinese languages other than Mandarin is restricted, children are forced to use Mandarin in school, all official business is done in Mandarin, etc. There is a concerted effort to make other languages economically unviable, and generally to disempower and discourage regional / minority cultures. The grandparent poster’s experience is evidence that this strategy is working out. It is similar to the way the Chinese government assigns non-native political officials to rule each region, and severely censors any politically controversial communication/media. That is, it is yet another tool of authoritarian social control, an effort to forestall any political opposition to the central government and its unresponsive top-down decision-making process. India does not have the same kind of authoritarian governing institutions, so similar forced homogenization would not be politically viable.
    - baobrain 9y ago
      > the Chinese government doesn’t want to acknowledge that they are separate mutually unintelligible languages. Except they are not mutually unintelligible. Put someone from heilongjiang province in Sichuan and they will still be able to understand the language, albeit with more difficulty. Though there are dialects that do have completely different pronunciation, they all use the same underlying script, save a select few minority languages. Mandarin Chinese is taught in school, but everyone still uses the local dialect to speak with each other. I'm not even denying the CCP has ulterior motives in doing this, but your original claim was simply incorrect and disingenuous.
      - krapht 9y ago
        Are you a native speaker? Because 廣東話 and 閩南話 are totally unintelligible to me, and I want your language superpower.
        HumanDrivenDev 9y ago
        Its just typical chinese nationalist mythology. 'We all speak the same language' is just another fairy tale they drill into their heads, along with '5000 years of culture' When youve talked to one of them youve talked to all of them.
        tonylinn80 9y ago
        A few dialects are unintelligible indeed, and they're mostly spoken in southern parts of China. But most people can understand dialects spoken in central and east (north of Changjiang river) fairly easily even though they cannot speak those dialects.
        HumanDrivenDev 9y ago
        And Portuguese people can often understand Spanish. Are they dialects?
      - 9y ago
  - PeterisP 9y ago
    That's how languages form. You don't get a language like French just by hoping for it to emerge or by letting dialects run their own lives and continue to diverge; you take it by taking a bunch of Romance dialects including quite separate ones (Langue d'Oc and the Langue d'Oil groups) and pushing them together through a common system of mass media (printed for the time, but still), education and cultural acceptance of a "proper dialect" at the top end of the society. That's how you get a strong language that helps you to unify a country and reduce internal barriers of communication; and that's what the Chinese are doing.
    - titanix2 9y ago
      I never heard that the language was standardized taking langue d’oc elements or any other French languages or dialects. You should give some sources to affirm that. The standard French is based on the French spoken in Paris which was then extended (or imposed in some case) in the whole country via the administration and the public school (forbidding the use of dialects).
    - _emacsomancer_ 9y ago
      By "that's how languages form", I assume you mean something like "standardized languages", which then becomes somewhat circular, because it's certainly not how languages form generally (and even standardized languages can form without mass media, and certainly did, even pre-writing). That's also not how standard French formed. Standardized French is Parisian French, so just the 'dialect' of the politically-important centre. Not too dissimilar to Latin (which was originally narrowly the dialect of Rome) in that. Whatever the propaganda, China still contains a number of non-mutually intelligible (though related) languages, but of course Mandarin enjoys much prominence and has an enormous number of speakers.
  - echevil 9y ago
    While this might be true for spoken language, luckily the Chinese writing system does represent how the the words are pronounced. All of these "dialects" share the same writing system. So it's much easier for someone speaking one of these dialects to learn and use Mandarin.
nkoren 9y ago
How does phonological complexity affect these things? I'm not a linguist, but Hindi at least seems way complex, with a much larger phoneme inventory than other languages I've studied.
- squiggleblaz 9y ago
  In the World Atlas of Language Structures http://wals.info/chapter/1 http://wals.info/chapter/1 , Hindi is considered to have a large consononant inventory. Hindi consonants plus vowels is not hugely different than English. But even though it's relatively large, I certainly wouldn't describe it as complex. It's mostly just the same properties repeated in different locations. Language areas aren't made more complex by additional phonological complexity. The question you've asked seems to be "Does phonological complexity cause more language diversity". When put this way, there doesn't seem to be any causal mechanism that could do it. For instance, one might say: Well, English has a lot of vowels. People in California might simplify them one way, whereas people in Texas might simplify them in another way so that they can't understand each other well: therefore, you get additional linguistic diversity. But this requires the Texans and the Californians to be isolated from each other which isn't what people mean when they say "India is a very linguistically diverse country". If we try it the other way "Does language diversity cause more phonological complexity", languages tend towards each other in case of diversity (because a person who speaks both Chinese and English will sometimes adopt features of one language into the other). This can sometimes lead to the propagation of more sounds (for instance, languages pretty much only use clicks if they're in contact with other languages that use clicks). And sometimes it can lead the elimination of them. I'm not sure of any particular research about this question, but my guess is on average it would tend towards the average, but if you took English and put it in India it wouldn't be too long before English in India sounds a lot more Indian.
- moultano 9y ago
  I don't think Hindi's phonology is more complex than English, but it is all explicit in the character set rather than implicit in the etymology. (The voicing difference between "this" and "thistle" is my favorite example of this, even beyond the fact that we're representing one of the most common consonant sounds in the language with two characters.)

azurezyq 9y ago

China does have very diverse dialects. But Mandarin is taught in all schools in mainland China and we treat it as standard. Also it is the unified way people from different area communicate. For younger generation this is not a problem, back in my college life where people in the same class came from different areas, from Xinjiang to Canton to the northeast corner. We have no problem understanding each other though sometimes funny with unique words or accents. It seems that the problem is not as severe in China as in India.

_emacsomancer_ 9y ago
And what are called "dialects" in the Chinese context are just as much dialects as French, Italian, and Romanian are "dialects" of Romance.
- jacobolus 9y ago
  These mostly get called “dialects” for political reasons: the Chinese government doesn’t want to acknowledge that they are separate mutually unintelligible languages. Media in Chinese languages other than Mandarin is restricted, children are forced to use Mandarin in school, all official business is done in Mandarin, etc. There is a concerted effort to make other languages economically unviable, and generally to disempower and discourage regional / minority cultures. The grandparent poster’s experience is evidence that this strategy is working out. It is similar to the way the Chinese government assigns non-native political officials to rule each region, and severely censors any politically controversial communication/media. That is, it is yet another tool of authoritarian social control, an effort to forestall any political opposition to the central government and its unresponsive top-down decision-making process. India does not have the same kind of authoritarian governing institutions, so similar forced homogenization would not be politically viable.
  - baobrain 9y ago
    > the Chinese government doesn’t want to acknowledge that they are separate mutually unintelligible languages. Except they are not mutually unintelligible. Put someone from heilongjiang province in Sichuan and they will still be able to understand the language, albeit with more difficulty. Though there are dialects that do have completely different pronunciation, they all use the same underlying script, save a select few minority languages. Mandarin Chinese is taught in school, but everyone still uses the local dialect to speak with each other. I'm not even denying the CCP has ulterior motives in doing this, but your original claim was simply incorrect and disingenuous.
    - krapht 9y ago
      Are you a native speaker? Because 廣東話 and 閩南話 are totally unintelligible to me, and I want your language superpower.
      - HumanDrivenDev 9y ago
        Its just typical chinese nationalist mythology. 'We all speak the same language' is just another fairy tale they drill into their heads, along with '5000 years of culture' When youve talked to one of them youve talked to all of them.
        tonylinn80 9y ago
        A few dialects are unintelligible indeed, and they're mostly spoken in southern parts of China. But most people can understand dialects spoken in central and east (north of Changjiang river) fairly easily even though they cannot speak those dialects.
        HumanDrivenDev 9y ago
        And Portuguese people can often understand Spanish. Are they dialects?
    - 9y ago
- PeterisP 9y ago
  That's how languages form. You don't get a language like French just by hoping for it to emerge or by letting dialects run their own lives and continue to diverge; you take it by taking a bunch of Romance dialects including quite separate ones (Langue d'Oc and the Langue d'Oil groups) and pushing them together through a common system of mass media (printed for the time, but still), education and cultural acceptance of a "proper dialect" at the top end of the society. That's how you get a strong language that helps you to unify a country and reduce internal barriers of communication; and that's what the Chinese are doing.
  - titanix2 9y ago
    I never heard that the language was standardized taking langue d’oc elements or any other French languages or dialects. You should give some sources to affirm that. The standard French is based on the French spoken in Paris which was then extended (or imposed in some case) in the whole country via the administration and the public school (forbidding the use of dialects).
  - _emacsomancer_ 9y ago
    By "that's how languages form", I assume you mean something like "standardized languages", which then becomes somewhat circular, because it's certainly not how languages form generally (and even standardized languages can form without mass media, and certainly did, even pre-writing). That's also not how standard French formed. Standardized French is Parisian French, so just the 'dialect' of the politically-important centre. Not too dissimilar to Latin (which was originally narrowly the dialect of Rome) in that. Whatever the propaganda, China still contains a number of non-mutually intelligible (though related) languages, but of course Mandarin enjoys much prominence and has an enormous number of speakers.
- echevil 9y ago
  While this might be true for spoken language, luckily the Chinese writing system does represent how the the words are pronounced. All of these "dialects" share the same writing system. So it's much easier for someone speaking one of these dialects to learn and use Mandarin.

nkoren 9y ago

How does phonological complexity affect these things? I'm not a linguist, but Hindi at least seems way complex, with a much larger phoneme inventory than other languages I've studied.

squiggleblaz 9y ago
In the World Atlas of Language Structures http://wals.info/chapter/1 http://wals.info/chapter/1 , Hindi is considered to have a large consononant inventory. Hindi consonants plus vowels is not hugely different than English. But even though it's relatively large, I certainly wouldn't describe it as complex. It's mostly just the same properties repeated in different locations. Language areas aren't made more complex by additional phonological complexity. The question you've asked seems to be "Does phonological complexity cause more language diversity". When put this way, there doesn't seem to be any causal mechanism that could do it. For instance, one might say: Well, English has a lot of vowels. People in California might simplify them one way, whereas people in Texas might simplify them in another way so that they can't understand each other well: therefore, you get additional linguistic diversity. But this requires the Texans and the Californians to be isolated from each other which isn't what people mean when they say "India is a very linguistically diverse country". If we try it the other way "Does language diversity cause more phonological complexity", languages tend towards each other in case of diversity (because a person who speaks both Chinese and English will sometimes adopt features of one language into the other). This can sometimes lead to the propagation of more sounds (for instance, languages pretty much only use clicks if they're in contact with other languages that use clicks). And sometimes it can lead the elimination of them. I'm not sure of any particular research about this question, but my guess is on average it would tend towards the average, but if you took English and put it in India it wouldn't be too long before English in India sounds a lot more Indian.
moultano 9y ago
I don't think Hindi's phonology is more complex than English, but it is all explicit in the character set rather than implicit in the etymology. (The voicing difference between "this" and "thistle" is my favorite example of this, even beyond the fact that we're representing one of the most common consonant sounds in the language with two characters.)