23 ms·

Why is NLP these days equivalent of Natural English Language Processing? There are much more languages in the world.

paradite 10y ago
I am surprised to see how an off-topic discussion generated much more comments than the discussion on this cool project per se. It seems to be the norm these days on HN...
- rolfvandekrol 10y ago
  Surpises me too. I just pointed out that the project at hand hardly even mentions that it is targeted at English, as if NLP is English by definition and that doesn't need any explanation.
tfm 10y ago
Short answer would be availability of tagged data, in turn dependent on historical research, which is a laggy history of who's been funding the research. This is changing, thankfully; check out e.g. the European Language Resources Association's catalogues, http://catalog.elra.info/ http://catalog.elra.info/ or any of the very incomplete list on wikipedia: https://en.wikipedia.org/wiki/List_of_text_corpora https://en.wikipedia.org/wiki/List_of_text_corpora or in fact the various international Wikipediae themselves! As for this particular project being English only ... itch to scratch probably ;-)
- danieldk 10y ago
  Short answer would be availability of tagged data, Part-of-speech and syntax-annotated corpora have been available for a long time for many other languages than English. E.g. for German: http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/tiger.html http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora... http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueba-dz.html http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueb... Dutch: https://www.let.rug.nl/vannoord/Lassy/ https://www.let.rug.nl/vannoord/Lassy/ http://lands.let.ru.nl/cgn/ http://lands.let.ru.nl/cgn/ https://www.let.rug.nl/vannoord/trees/ https://www.let.rug.nl/vannoord/trees/ French: http://www.llf.cnrs.fr/en/Gens/Abeille/French-Treebank-fr.php http://www.llf.cnrs.fr/en/Gens/Abeille/French-Treebank-fr.ph... I think that one of the problems is that NLP research of English typically has a higher chance of getting accepted at conferences.
paulddraper 10y ago
*many more languages
jnky 10y ago
On a related note: why are we putting up with other languages anyway? I've been asking myself for a long time why we (as in humanity) don't try to standardize on a language. The language of the Internet is English for the most part, and since the Internet is the greatest facilitator of communication in human history, that standard language might as well be English. I firmly believe the world would be a much better place if people from everywhere could communicate with each other without being inhibited by the language barrier. I understand that this is not something that can be done overnight, but a lot of people learn English as a second langue today, so maybe a switch could be made in a generation or two, as people who don't speak English as a first or second language slowly die out.
- mpeg 10y ago
  I can't disagree more. The language of _your_ internet is English. Personally it always bothers me to think of how the largest parts of the internet are essentially walled off by language. China has essentially its own version of everything we use day to day, so does Japan, Russia, LATAM ...
  - jnky 10y ago
    But that's exactly my point. Instead of having a bunch of barely connected Internets walled off by language, why not standardize language and have one global venue of communication, where everyone can work together? I am German and I often come across this with my friends, all of whom speak English as a second language. Yet they consume mostly the German parts of the Internet out of habit and convenience. But as far as I can tell, they are worse off for it, since they don't have access to a lot of the stuff the Internet has to offer. That is especially obvious when searching for IT/tech/science topics, but also relevant in other contexts.
    - Matt3o12_ 10y ago
      I couldn't agree more! As a German, all my friends always ask me why my phone is in English and they just won't understand how much more the "English Internet" has to offer. Just the other day, we had to look up something for our school and I simply sumerized the first few lines of the English Wikipedia article and my teacher was amazed how quickly I got those information. Furthermore, English allows you to see topics from many different views – people from all over the world express their opinions which is much more then most German Jornalist find relevant/agree with.
dreamsofdragons 10y ago
If I were to get back into it, I'd still pick English because English is lingua franca. But I fully suspect that there are equal efforts in Madarin, Hindi, Spanish and French, we just don't hear a lot about them as outsiders of the community.
nkrisc 10y ago
Probably related to English being the third most spoken language in the world and widely represented in international scientific communities, even by non-native speakers. Not defending it.
stephenr 10y ago
I used the Stanford nlp in a project last year, apparently it can (with appropriate models) handle German, Chinese (not sure which flavour) and Japanese (I think?). Unfortunately what I needed to process was likely to be in either English or Indonesian, without forewarning which it was, so we had to use an api (we used Google in the end , but the Microsoft one worked too) to detect + translate the text to English, which has some interesting side affects when dealing with numbers with suffixes, and dates, due to limitatioms of what Stanford nlp recognised (1 million is recognised as 1000000, 1M is not) Also, handling informal text can be a nightmare - I'm still wondering if a mechanical Turk solution wouldn't have been better.

I am surprised to see how an off-topic discussion generated much more comments than the discussion on this cool project per se. It seems to be the norm these days on HN...

rolfvandekrol 10y ago
Surpises me too. I just pointed out that the project at hand hardly even mentions that it is targeted at English, as if NLP is English by definition and that doesn't need any explanation.

tfm 10y ago

Short answer would be availability of tagged data, in turn dependent on historical research, which is a laggy history of who's been funding the research. This is changing, thankfully; check out e.g. the European Language Resources Association's catalogues, http://catalog.elra.info/ http://catalog.elra.info/ or any of the very incomplete list on wikipedia: https://en.wikipedia.org/wiki/List_of_text_corpora https://en.wikipedia.org/wiki/List_of_text_corpora or in fact the various international Wikipediae themselves! As for this particular project being English only ... itch to scratch probably ;-)

danieldk 10y ago
Short answer would be availability of tagged data, Part-of-speech and syntax-annotated corpora have been available for a long time for many other languages than English. E.g. for German: http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/tiger.html http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora... http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueba-dz.html http://www.sfs.uni-tuebingen.de/ascl/ressourcen/corpora/tueb... Dutch: https://www.let.rug.nl/vannoord/Lassy/ https://www.let.rug.nl/vannoord/Lassy/ http://lands.let.ru.nl/cgn/ http://lands.let.ru.nl/cgn/ https://www.let.rug.nl/vannoord/trees/ https://www.let.rug.nl/vannoord/trees/ French: http://www.llf.cnrs.fr/en/Gens/Abeille/French-Treebank-fr.php http://www.llf.cnrs.fr/en/Gens/Abeille/French-Treebank-fr.ph... I think that one of the problems is that NLP research of English typically has a higher chance of getting accepted at conferences.

paulddraper 10y ago

*many more languages

jnky 10y ago

On a related note: why are we putting up with other languages anyway? I've been asking myself for a long time why we (as in humanity) don't try to standardize on a language. The language of the Internet is English for the most part, and since the Internet is the greatest facilitator of communication in human history, that standard language might as well be English. I firmly believe the world would be a much better place if people from everywhere could communicate with each other without being inhibited by the language barrier. I understand that this is not something that can be done overnight, but a lot of people learn English as a second langue today, so maybe a switch could be made in a generation or two, as people who don't speak English as a first or second language slowly die out.

mpeg 10y ago
I can't disagree more. The language of _your_ internet is English. Personally it always bothers me to think of how the largest parts of the internet are essentially walled off by language. China has essentially its own version of everything we use day to day, so does Japan, Russia, LATAM ...
- jnky 10y ago
  But that's exactly my point. Instead of having a bunch of barely connected Internets walled off by language, why not standardize language and have one global venue of communication, where everyone can work together? I am German and I often come across this with my friends, all of whom speak English as a second language. Yet they consume mostly the German parts of the Internet out of habit and convenience. But as far as I can tell, they are worse off for it, since they don't have access to a lot of the stuff the Internet has to offer. That is especially obvious when searching for IT/tech/science topics, but also relevant in other contexts.
  - Matt3o12_ 10y ago
    I couldn't agree more! As a German, all my friends always ask me why my phone is in English and they just won't understand how much more the "English Internet" has to offer. Just the other day, we had to look up something for our school and I simply sumerized the first few lines of the English Wikipedia article and my teacher was amazed how quickly I got those information. Furthermore, English allows you to see topics from many different views – people from all over the world express their opinions which is much more then most German Jornalist find relevant/agree with.

dreamsofdragons 10y ago

If I were to get back into it, I'd still pick English because English is lingua franca. But I fully suspect that there are equal efforts in Madarin, Hindi, Spanish and French, we just don't hear a lot about them as outsiders of the community.

nkrisc 10y ago

Probably related to English being the third most spoken language in the world and widely represented in international scientific communities, even by non-native speakers. Not defending it.

stephenr 10y ago

I used the Stanford nlp in a project last year, apparently it can (with appropriate models) handle German, Chinese (not sure which flavour) and Japanese (I think?). Unfortunately what I needed to process was likely to be in either English or Indonesian, without forewarning which it was, so we had to use an api (we used Google in the end , but the Microsoft one worked too) to detect + translate the text to English, which has some interesting side affects when dealing with numbers with suffixes, and dates, due to limitatioms of what Stanford nlp recognised (1 million is recognised as 1000000, 1M is not) Also, handling informal text can be a nightmare - I'm still wondering if a mechanical Turk solution wouldn't have been better.