10 ms·

Do NLP Beyond English

148 points by dsr12 6y ago

laGrenouille 6y ago
I generally agree with point made in the article that too much NLP research is focused only on English and a small number of other high-resourced languages. To me, this is part of a larger problem with natural language processing's obsession with "state-of-the-art" metrics and general abandonment of broader research in linguistics. The situation for those who, like myself, work in applied linguistics is not actually so dire... At least when working with the languages that have a reasonable amount of training data. Decent enough treebanks exist for lemmatisation, POS-tagging, and dependency parsing for dozens of languages [0]. Fast tools such as spaCy (15 languages) [1] and udpipe (40+ languages) [2] are freely available and work well for most applied tasks. There are even decent word embeddings available trained on various versions of Wikipedia [3]. Of course, some of the issue is that these very tasks are biased towards the specific structures of Indo-European languages. However, for getting work done (building sentiment classifiers, document clustering, NER, ect.), currently available tools make it possible to work with a large proportion of the currently available data. There is still a lot of work to be done own regards to computational research on non-English languages, but a lot of the problem right now is recognition of applied work by the top NLP conferences rather than a complete lack of quality work currently being done. [0] https://universaldependencies.org https://universaldependencies.org [1] https://spacy.io/models https://spacy.io/models [2] https://ufal.mff.cuni.cz/udpipe/models https://ufal.mff.cuni.cz/udpipe/models [3] https://github.com/facebookresearch/fastText/blob/master/docs/crawl-vectors.md https://github.com/facebookresearch/fastText/blob/master/doc...
- dhairya 6y ago
  One thing i think about is what where would NLP and linguistics research gone if another language (other than english) was the predominant language. Granted there is quite of bit interesting fundamental research in other countries and languages. But it seems many of problems and approaches developed in language understanding and NLP target specific idiosyncracies of the English language. If NLP researched with say German (which has a deep and overly explicit vocabulary) or Hindi or Mandarian, would the fundamental NLP approaches and problem areas be different and potentially better?
  - TomMarius 6y ago
    I recommend you to check out NLP research regarding the Czech language. It has high quality researchers keen on making use of the language's features (it's a highly regular and expressive grammar). Sadly not much research is done, but what is done is interesting.
  - YeGoblynQueenne 6y ago
    Agreed. The way I see it, current research directions have focused on approaches that work best with large datasets because such datasets exist for English, and because English just happens to be a language that is possible to represent well by training on large datasets. It helps of course that English is the de facto lingua franca of most of the technologically developed world (if not the rest of the world also). If another language was the lingua franca, current approaches would not work so well. The article makes a point about how we assume that pre-trained embeddings encode all relevant information about a language when this may not be the case for arbitrary languages. The article also makes a point about the morphological simplicity of English which is of course a great asset for a lingua franca, and also simplifies the language modelling task. Imagine if the lingua franca was, e.g. ancient Greek (as it was, in ancient times, all around the Mediterrannean) or something with similar morphological intricacies (and Greek is still an Indo-European language; I just don't know much about, e.g. Finno-Ugric to give an example about them).
- tlb 6y ago
  Is it practical to do NLP research using a language you're not fluent in? Like, does most research require the experimenter to be able to judge fluency and ambiguity themselves?
  - empiko 6y ago
    For sure. Many NLP scientific benchmarks are multilingual today and you need to evaluate on X languages. You are of course not expected to be fluent in all of them. Most of the deep learning based approaches are quite language-universal, i.e. you have a pre-trained language model or word embeddings and you just slap a sequence modeling model on top of that and you let the neural network do the magic. However, you can still incorporate language-specific knowledge if you want to get the best performance for that one particular language. This is mainly true for academic research. If you are developing a rule-based or keyword-based application (which is still often the case outside of academia), it can actually help to be fluent to get the rules right.
YeGoblynQueenne 6y ago
>> Recent models have repeatedly matched human-level performance on increasingly difficult benchmarks—that is, in English using labelled datasets with thousands and unlabelled data with millions of examples. In the process, as a community we have overfit to the characteristics and conditions of English-language data. In particular, by focusing on high-resource languages, we have prioritised methods that work well only when large amounts of labelled and unlabelled data are available. This is an interesting observation. Training on large datasets tends to be framed as a strength, e.g. we have recently seen articles praising OpenAI's GPT-3 for being, well, big. The truth is that model size (and the assorted dataset size) is a bug, not a feature. It is the result of a dearth of research on approaches with low sample complexity. Or in other words, it's the result of consistently picking the low-hanging fruit and calling sour grapes on any domain for which there isn't sufficiently large data ("who cares about Swahilli?"). >> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application. And this is interesting to read in the context of the recent manic hype about GPT-3, itself described in a paper titled "Language models are few-shot learners", hilariously attempting to sweep under the carpet the amount of data and compute required to get to the point where a language model can do "few" shot learning. In general, the excitement about the successes that have come from training large, deep neural nework models with very large datasets has only served to avoid posing the obvious question about such data-hungry approaches: what do we do when there isn't a lot of data? Human language with all its minutely fragmented diversity, turns out to be just that kind of domain.
- ShamelessC 6y ago
  I see this argument a lot. I have essentially zero ml experience, but isn't it fair to say that DNA indeed carries a similarly large amount of data? Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off. Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.
YeGoblynQueenne 6y ago
>> English and the small set of other high-resource languages are in many ways not representative of the world's other languages. Many resource-rich languages belong to the Indo-European language family, are spoken mostly in the Western world, and are morphologically poor, i.e. information is mostly expressed syntactically, e.g. via a fixed word order and using multiple separate words rather than through variation at the word level. I thought I'd give an example of this for readers who don't speak many languages other than English. My native language is Greek, which is an Indo-European language, but it has some morphological intricacies that make it more complicated to learn and use (as a second language and from what I'm told of course) than English. For example, in Greek, nouns have a gender [1]: masculine, feminine or neuter. For instance, "the dog" is "ο σκύλος" ("the male dog") while "the cat" is "η γάτα" (the female cat). However, grammmatical gender is not necessarily fixed so one can also say "η σκύλα" ("the female dog") or "ο γάτος" ("the male cat"). So there is a word-root ("σκυλ-" for dog, "γατ-" for cat) that is then modified by a termination, typically -ος, -η -ο for the nominative of each gender. The terminations change depending on the declention, for example the genetive for "σκύλος" is "σκύλου" and the generative for "σκύλα" is "σκύλας", etc. The termination also changes to denote number, singular or plural (ancient Greek also used to have a dual number, used to refer to pairs of nouns). Terminations vary depending on number according to gender and declention. So for example the plural of "σκύλος" is "σκύλοι", genetive "σκύλων" and the plural of "σκύλα" is "σκύλες", genetive "σκυλών". Compare this with English where a dog is a dog is a dog, there is only one plural form, "dogs" and there is only one genetive form "dog's" or "dogs'" for each number. And a female dog is either a circumlocution, "female dog" or an entirely new word, "bitch", with its own simple set of transformations for number and genitive "bitches" and "bitch's" or "bitches'". Basically, changing meaning in English can be represented by adding a few words to a vocabulary- but changing meaning in Greek requires manipulating words at the structural level. There are probably a few counter-examples to the above but my understanding is that this is how it works for the most part. And this, in particular the abilty of English to represent more meanings with more words, might go some way to explain why approaches that work best with large datasets tend to be favoured in English NLP. Edit: not an NLP expert or a linguist so corrections welcome, of course! ______________ [1] Other Indo-European languages also have genders but the interesting thing is that those genders don't always match, between languages. This is an endless source of confusion for foreign language learners. For example I speak French from an early age and I only realised much later in life that Greek and French do not always use the same gender for the same nouns, even though I'd been using the right genders in my speech. For example, both "dog" and "cat" are masculine in French: "le chien" and "le chat". But while I always used "le chat" correctly, I always thought of it in my mind as "η γάτα", the feminine Greek noun. Human language is weird!
jaimex2 6y ago
THe moneys all in English speaking countries and China so... do NLP in Mandarin too.
PeterStuer 6y ago
The cheap money, as in Oligarchy backed petrodollar slush funds, is residing mainly in the US. The 'defacto' second language of the world is English. Combine those two and it is not hard to see why English NLP is so dominant. I'm not yet sure whether that is a good or a bad thing for non-English-as-a-first-languege regions.
- ShamelessC 6y ago
  I don't have a horse in this race, but surely it could only be bad for those regions, right? There's a whole world of applications involving NLP which non-English speakers don't have access to.
  - PeterStuer 6y ago
    On the other hand, a huge amount of those applications target consumers to get them to misspend more money, so not having them might be a benefit? Surveillance capitalism relies on getting ever more info about you, and exploiting it or selling it on to all that want to exploit it. NLP is one vector to extract that info from raw data.
eukgoekoko 6y ago
It's not just NLP, English is pretty bad as an intermediate language for translations from language A to language B. If I try translating a Russian word "пружина" ("a mechanical spring") to German using Google translate, I end up with "Frühling", which in fact means "springtime". This is an obvious artefact of transitive translatiion: Russian -> English -> German. Providing context may help, but still translation to English strips important pieces of information.
- frankie_t 6y ago
  I don't think this is inherent to English, or any other language (perhaps in more specific cases when there is no word with the similar meaning). I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
  - Shorel 6y ago
    I think it is inherent to English at least in degree. Reading translated books from Polish to Spanish or Russian to Spanish conveys a lot more information, than reading the same book in their English translation. It's like every subtle nuance is lost in English.
    - frankie_t 6y ago
      There could be multiple factors I think, like: your language skill level, translation quality, "closeness" of languages, psychological predisposition towards your native language. Depending on the form of the thing you are translating it can be simply impossible to translate properly (like poetry). I'm currently reading English translation of "Black Obelisk" which I believe is written in German and it isn't any worse than Russian translation (my native language) to me. In any case, what was originally asserted is that English is somehow worse than other languages as a "transitional" translation language for words or simple phrases, so I argued with that specific idea. Translating literary works is a subject of it's own and where the quality is much harder to measure.
marcinzm 6y ago
> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down. > Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application. This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
- thomasahle 6y ago
  > I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from. I mostly agree with you, but it does seem like current models require more data to learn language than baby humans. This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
  - slx26 6y ago
    But baby humans tend to associate phonetic symbols (spoken words) to what they are seeing, and the learning method is very different. Their context is much broader, helped by multiple senses. NLP is extremely contextless compared to what humans do. You can't talk about "more data" so easily.
  - 6y ago
toolslive 6y ago
My native language is Dutch. Google handles it poorly. As a result people have adapted to either search in English, or to morph the original search (that google does not understand) into sometime that has a higher changes of yielding good results. I'm pretty sure Dutch isn't the only language that fares this way. Now the main reason (at least, I think it is) it's handled poorly is that Dutch has an infinite number of words ( * ) and order in compositions matters a lot. This conflicts with the typical precomputations that try to reduce the number of words etc. ( * ) take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.
- dtech 6y ago
  Google does a pretty decent job for me in Dutch; Duckduckgo and Bing are awful to the point of being useless. Don't agree with your hypothesis either, results are bad even for short simple sentences with common non-combined nouns. They aren't that common in Dutch anyway unless they have evolved into a more or less standalone noun. Dutch nor English people really think of a flamethrower as a thrower of flames, it's just another noun.
- yorwba 6y ago
  > take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too. This is also how it works in English, except the concatenation may be written with a space as "A B". If anything, writing it as "AB" should make search more accurate, since you're less likely to get results with "BA" than getting results with "B A" when searching for "A B".
  - Tainnor 6y ago
    Only if your NLP models actually decompose words. Simpler ones don't. But while it's technically true that English also has compounds, it just writed them apart, it is also the case that German (don't know about Dutch) uses compounds, and especially long compounds with more than two nouns, a lot more than English, where you might more often use a whole sentence.
azangru 6y ago
I am not a native English-speaker, but my dream is that the world gradually converges on a single standard for international communication, and that this standard is English, so that an ever-growing proportion of the population will become fluent in it. I understand various reasons for why this is unlikely to happen; and I realise that there've been lots of centrifugal forces lately that make this vision ever less likely; but I can't help but smile inwardly when I hear that NLP is so much focused on English, and can't help rooting for it to remain so.
- bitL 6y ago
  I think this is a horrible idea; unifying the mindsets of all people via the same language would mean that we might get forever stuck in some local optimum and there won't be any other culture that could push humanity forward. People tend to become complacent if they aren't feeling competition, leading to a downfall, and that is true for all aspects of life. It's like we are all in some sort of Reinforcement Learning algorithm and unifying the language portion would significantly restrict the exploration phase, the one that moves world forward. There are many concepts unique to each language, different emphasis on different things, different emotional response to different sentences, leading to unique ways of exploring the universe, and that all would be lost.
  - azangru 6y ago
    > unifying the mindsets of all people via the same language There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language. To me, the prospect of cultures having to communicate with each other via such a broken mechanism as translation is far more repugnant than a linguistically unified superculture within which ideas flow unimpeded. The more language users there are, the more capable of expression the language will become; so no, I am not particularly worried about getting stuck in a local maximum.
    - djedr 6y ago
      I think it goes both ways: language shapes mind and mind shapes language. It's a feedback loop.
LockAndLol 6y ago
Though i agree with the main point, the blog post went off on a tangent about how some other languages are in danger because of English. IMO if the speakers of the language do not: 1. Provide materials in their own language 2. Consume materials in their own language 3. Actively and exclusively contribute to the world in English it is already a statement about the necessity of their language: it isn't. Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? If its own speakers kill the language (actively or passively) who are we to say they shouldn't?
- slx26 6y ago
  > Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? Because rational choices at an individual level do not necessarily align with rational choices at a social collective level. "Meditations on Moloch" is a good read on this topic. You can easily see how collectively the language diversity is positive, but individually it kinda makes sense to bet for the biggest language instead. And to be fair, there's a lot of people investing in their own languages, just not enough, or not with enough power.
- yorwba 6y ago
  I assume you're referring to this paragraph: A continuing lack of technological inclusion will not only exacerbate the language divide but it may also drive speakers of unsupported languages and dialects to high-resource languages with better technological support, further endangering such language varieties. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, to lower language and literacy barriers, we need to apply our models to non-English languages. Have you considered what this means, concretely, for a speaker of an endangered language? Suppose I want to "invest" into my language and 1. Provide materials: - Written? I need some kind of keyboard support. In the worst case, this might entail getting new characters added to Unicode. Or I could just use paper, but then I lose out on all the benefits of digital text editing. - Spoken/Signed? In some ways, that's easier, since I can just record myself. But if I want closed captions or some kind of transcript, I can't make use of automatic tools, I have to transcribe it completely manually. 2. Consume materials: - But how do I find them? What if searching for words in my language mostly turns up results in other languages because the search engine assumes I misspelled my query? - It might happen that the text I'm trying to read uses characters that were considered "too rare" to be included in the default font on my device. On most versions of Android, it's not possible to install a new font to fix that. I'm not sure about other mobile operating systems. Those are problems that don't require speakers of the language to do something differently (such as just using English instead...) but instead need technical solutions that can very well be implemented by someone who doesn't speak the language. I don't speak an endangered language myself, but I've written code to support dozens of languages I don't speak, because someone who did speak the language asked for a particular change that would make life easier for them.
627467 6y ago
I'm reading lots of comments reacting against the premise of the article in name of some kind of efficiency or inherent value of a universal standar. I wonder if commenters have considered that Research is useful to understand the present and more importantly prepare for the future. While English being useful língua Franca today seems to back decision to just focus on English research the point is: will this English uptake continue at same rate in future? Isn't there some kind of bubble effect where here in hn many think that (as non English native speakers) we can all engage so we think English is the only useful thing for us to understand? What about all possible unknown Mandarin NLP research (which I think many agree) may be useful for non-mandarin speakers? Also, isn't one of the points of Information Technology to enable/create value in a long tail of experiences? Focusing on a single attribute of a substantial part of the world seems counter productive and shortsighted...
tanilama 6y ago
With new direction of research, like pre-taining and subword vocab, the models now are mostly language agnostic. English or not, it is just sequence of tokens. Difference mainly lies in data abundance, where the distribution is severly skewed.
mindvirus 6y ago
Can anyone chime in to how Chinese NLP compares? I speak Mandarin as a second language at an intermediate level, and from my perspective it has some interesting properties compared to English (my native language) that seem like they might make NLP easier in ways that English NLP might be hard: - The grammar for standard Mandarin is much simpler than English. No verb conjugations or strange tense rules ("go" -> "went")! - Tense and even voice are explicit and additive - i.e. adding "呢" or "吧" to a sentence. - Written, it seems less ambiguous. i.e. "duck" or "grave" have different meanings in English based on context, but that seems much much rarer in Chinese - two words spoken the same way but with different meanings usually have different written forms.
- Tainnor 6y ago
  Mandarin has even less morphology than English[1], so to some extent, working on Mandarin would not "fix" some of the issues of NLP models often performing poorly on morphology-rich languages. But it's possible that focusing more on Mandarin would still lead to improvements in other areas, because the two languages are otherwise quite different (I wonder how well speech recognition works for tonal languages, for example?). [1] Possibly not without reason: there is some indication that languages spoken by a large number of speakers tend to become structurally simpler, while very small isolated languages can often develop surprising complexity.
bfung 6y ago
A cynical take supporting the above: NLP in other languages can help sell better ads in those languages. You all can follow that chain of thought down to it's corporate conclusion =)

148 points by dsr12 6y ago

I generally agree with point made in the article that too much NLP research is focused only on English and a small number of other high-resourced languages. To me, this is part of a larger problem with natural language processing's obsession with "state-of-the-art" metrics and general abandonment of broader research in linguistics. The situation for those who, like myself, work in applied linguistics is not actually so dire... At least when working with the languages that have a reasonable amount of training data. Decent enough treebanks exist for lemmatisation, POS-tagging, and dependency parsing for dozens of languages [0]. Fast tools such as spaCy (15 languages) [1] and udpipe (40+ languages) [2] are freely available and work well for most applied tasks. There are even decent word embeddings available trained on various versions of Wikipedia [3]. Of course, some of the issue is that these very tasks are biased towards the specific structures of Indo-European languages. However, for getting work done (building sentiment classifiers, document clustering, NER, ect.), currently available tools make it possible to work with a large proportion of the currently available data. There is still a lot of work to be done own regards to computational research on non-English languages, but a lot of the problem right now is recognition of applied work by the top NLP conferences rather than a complete lack of quality work currently being done. [0] https://universaldependencies.org https://universaldependencies.org [1] https://spacy.io/models https://spacy.io/models [2] https://ufal.mff.cuni.cz/udpipe/models https://ufal.mff.cuni.cz/udpipe/models [3] https://github.com/facebookresearch/fastText/blob/master/docs/crawl-vectors.md https://github.com/facebookresearch/fastText/blob/master/doc...

dhairya 6y ago
One thing i think about is what where would NLP and linguistics research gone if another language (other than english) was the predominant language. Granted there is quite of bit interesting fundamental research in other countries and languages. But it seems many of problems and approaches developed in language understanding and NLP target specific idiosyncracies of the English language. If NLP researched with say German (which has a deep and overly explicit vocabulary) or Hindi or Mandarian, would the fundamental NLP approaches and problem areas be different and potentially better?
- TomMarius 6y ago
  I recommend you to check out NLP research regarding the Czech language. It has high quality researchers keen on making use of the language's features (it's a highly regular and expressive grammar). Sadly not much research is done, but what is done is interesting.
- YeGoblynQueenne 6y ago
  Agreed. The way I see it, current research directions have focused on approaches that work best with large datasets because such datasets exist for English, and because English just happens to be a language that is possible to represent well by training on large datasets. It helps of course that English is the de facto lingua franca of most of the technologically developed world (if not the rest of the world also). If another language was the lingua franca, current approaches would not work so well. The article makes a point about how we assume that pre-trained embeddings encode all relevant information about a language when this may not be the case for arbitrary languages. The article also makes a point about the morphological simplicity of English which is of course a great asset for a lingua franca, and also simplifies the language modelling task. Imagine if the lingua franca was, e.g. ancient Greek (as it was, in ancient times, all around the Mediterrannean) or something with similar morphological intricacies (and Greek is still an Indo-European language; I just don't know much about, e.g. Finno-Ugric to give an example about them).
tlb 6y ago
Is it practical to do NLP research using a language you're not fluent in? Like, does most research require the experimenter to be able to judge fluency and ambiguity themselves?
- empiko 6y ago
  For sure. Many NLP scientific benchmarks are multilingual today and you need to evaluate on X languages. You are of course not expected to be fluent in all of them. Most of the deep learning based approaches are quite language-universal, i.e. you have a pre-trained language model or word embeddings and you just slap a sequence modeling model on top of that and you let the neural network do the magic. However, you can still incorporate language-specific knowledge if you want to get the best performance for that one particular language. This is mainly true for academic research. If you are developing a rule-based or keyword-based application (which is still often the case outside of academia), it can actually help to be fluent to get the rules right.

YeGoblynQueenne 6y ago

>> Recent models have repeatedly matched human-level performance on increasingly difficult benchmarks—that is, in English using labelled datasets with thousands and unlabelled data with millions of examples. In the process, as a community we have overfit to the characteristics and conditions of English-language data. In particular, by focusing on high-resource languages, we have prioritised methods that work well only when large amounts of labelled and unlabelled data are available. This is an interesting observation. Training on large datasets tends to be framed as a strength, e.g. we have recently seen articles praising OpenAI's GPT-3 for being, well, big. The truth is that model size (and the assorted dataset size) is a bug, not a feature. It is the result of a dearth of research on approaches with low sample complexity. Or in other words, it's the result of consistently picking the low-hanging fruit and calling sour grapes on any domain for which there isn't sufficiently large data ("who cares about Swahilli?"). >> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application. And this is interesting to read in the context of the recent manic hype about GPT-3, itself described in a paper titled "Language models are few-shot learners", hilariously attempting to sweep under the carpet the amount of data and compute required to get to the point where a language model can do "few" shot learning. In general, the excitement about the successes that have come from training large, deep neural nework models with very large datasets has only served to avoid posing the obvious question about such data-hungry approaches: what do we do when there isn't a lot of data? Human language with all its minutely fragmented diversity, turns out to be just that kind of domain.

ShamelessC 6y ago
I see this argument a lot. I have essentially zero ml experience, but isn't it fair to say that DNA indeed carries a similarly large amount of data? Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off. Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.

YeGoblynQueenne 6y ago

>> English and the small set of other high-resource languages are in many ways not representative of the world's other languages. Many resource-rich languages belong to the Indo-European language family, are spoken mostly in the Western world, and are morphologically poor, i.e. information is mostly expressed syntactically, e.g. via a fixed word order and using multiple separate words rather than through variation at the word level. I thought I'd give an example of this for readers who don't speak many languages other than English. My native language is Greek, which is an Indo-European language, but it has some morphological intricacies that make it more complicated to learn and use (as a second language and from what I'm told of course) than English. For example, in Greek, nouns have a gender [1]: masculine, feminine or neuter. For instance, "the dog" is "ο σκύλος" ("the male dog") while "the cat" is "η γάτα" (the female cat). However, grammmatical gender is not necessarily fixed so one can also say "η σκύλα" ("the female dog") or "ο γάτος" ("the male cat"). So there is a word-root ("σκυλ-" for dog, "γατ-" for cat) that is then modified by a termination, typically -ος, -η -ο for the nominative of each gender. The terminations change depending on the declention, for example the genetive for "σκύλος" is "σκύλου" and the generative for "σκύλα" is "σκύλας", etc. The termination also changes to denote number, singular or plural (ancient Greek also used to have a dual number, used to refer to pairs of nouns). Terminations vary depending on number according to gender and declention. So for example the plural of "σκύλος" is "σκύλοι", genetive "σκύλων" and the plural of "σκύλα" is "σκύλες", genetive "σκυλών". Compare this with English where a dog is a dog is a dog, there is only one plural form, "dogs" and there is only one genetive form "dog's" or "dogs'" for each number. And a female dog is either a circumlocution, "female dog" or an entirely new word, "bitch", with its own simple set of transformations for number and genitive "bitches" and "bitch's" or "bitches'". Basically, changing meaning in English can be represented by adding a few words to a vocabulary- but changing meaning in Greek requires manipulating words at the structural level. There are probably a few counter-examples to the above but my understanding is that this is how it works for the most part. And this, in particular the abilty of English to represent more meanings with more words, might go some way to explain why approaches that work best with large datasets tend to be favoured in English NLP. Edit: not an NLP expert or a linguist so corrections welcome, of course! ______________ [1] Other Indo-European languages also have genders but the interesting thing is that those genders don't always match, between languages. This is an endless source of confusion for foreign language learners. For example I speak French from an early age and I only realised much later in life that Greek and French do not always use the same gender for the same nouns, even though I'd been using the right genders in my speech. For example, both "dog" and "cat" are masculine in French: "le chien" and "le chat". But while I always used "le chat" correctly, I always thought of it in my mind as "η γάτα", the feminine Greek noun. Human language is weird!

jaimex2 6y ago

THe moneys all in English speaking countries and China so... do NLP in Mandarin too.

PeterStuer 6y ago

The cheap money, as in Oligarchy backed petrodollar slush funds, is residing mainly in the US. The 'defacto' second language of the world is English. Combine those two and it is not hard to see why English NLP is so dominant. I'm not yet sure whether that is a good or a bad thing for non-English-as-a-first-languege regions.

ShamelessC 6y ago
I don't have a horse in this race, but surely it could only be bad for those regions, right? There's a whole world of applications involving NLP which non-English speakers don't have access to.
- PeterStuer 6y ago
  On the other hand, a huge amount of those applications target consumers to get them to misspend more money, so not having them might be a benefit? Surveillance capitalism relies on getting ever more info about you, and exploiting it or selling it on to all that want to exploit it. NLP is one vector to extract that info from raw data.

eukgoekoko 6y ago

It's not just NLP, English is pretty bad as an intermediate language for translations from language A to language B. If I try translating a Russian word "пружина" ("a mechanical spring") to German using Google translate, I end up with "Frühling", which in fact means "springtime". This is an obvious artefact of transitive translatiion: Russian -> English -> German. Providing context may help, but still translation to English strips important pieces of information.

frankie_t 6y ago
I don't think this is inherent to English, or any other language (perhaps in more specific cases when there is no word with the similar meaning). I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
- Shorel 6y ago
  I think it is inherent to English at least in degree. Reading translated books from Polish to Spanish or Russian to Spanish conveys a lot more information, than reading the same book in their English translation. It's like every subtle nuance is lost in English.
  - frankie_t 6y ago
    There could be multiple factors I think, like: your language skill level, translation quality, "closeness" of languages, psychological predisposition towards your native language. Depending on the form of the thing you are translating it can be simply impossible to translate properly (like poetry). I'm currently reading English translation of "Black Obelisk" which I believe is written in German and it isn't any worse than Russian translation (my native language) to me. In any case, what was originally asserted is that English is somehow worse than other languages as a "transitional" translation language for words or simple phrases, so I argued with that specific idea. Translating literary works is a subject of it's own and where the quality is much harder to measure.

marcinzm 6y ago

> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down. > Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application. This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.

thomasahle 6y ago
> I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from. I mostly agree with you, but it does seem like current models require more data to learn language than baby humans. This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
- slx26 6y ago
  But baby humans tend to associate phonetic symbols (spoken words) to what they are seeing, and the learning method is very different. Their context is much broader, helped by multiple senses. NLP is extremely contextless compared to what humans do. You can't talk about "more data" so easily.
- 6y ago

toolslive 6y ago

My native language is Dutch. Google handles it poorly. As a result people have adapted to either search in English, or to morph the original search (that google does not understand) into sometime that has a higher changes of yielding good results. I'm pretty sure Dutch isn't the only language that fares this way. Now the main reason (at least, I think it is) it's handled poorly is that Dutch has an infinite number of words ( * ) and order in compositions matters a lot. This conflicts with the typical precomputations that try to reduce the number of words etc. ( * ) take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.

dtech 6y ago
Google does a pretty decent job for me in Dutch; Duckduckgo and Bing are awful to the point of being useless. Don't agree with your hypothesis either, results are bad even for short simple sentences with common non-combined nouns. They aren't that common in Dutch anyway unless they have evolved into a more or less standalone noun. Dutch nor English people really think of a flamethrower as a thrower of flames, it's just another noun.
yorwba 6y ago
> take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too. This is also how it works in English, except the concatenation may be written with a space as "A B". If anything, writing it as "AB" should make search more accurate, since you're less likely to get results with "BA" than getting results with "B A" when searching for "A B".
- Tainnor 6y ago
  Only if your NLP models actually decompose words. Simpler ones don't. But while it's technically true that English also has compounds, it just writed them apart, it is also the case that German (don't know about Dutch) uses compounds, and especially long compounds with more than two nouns, a lot more than English, where you might more often use a whole sentence.

azangru 6y ago

I am not a native English-speaker, but my dream is that the world gradually converges on a single standard for international communication, and that this standard is English, so that an ever-growing proportion of the population will become fluent in it. I understand various reasons for why this is unlikely to happen; and I realise that there've been lots of centrifugal forces lately that make this vision ever less likely; but I can't help but smile inwardly when I hear that NLP is so much focused on English, and can't help rooting for it to remain so.

bitL 6y ago
I think this is a horrible idea; unifying the mindsets of all people via the same language would mean that we might get forever stuck in some local optimum and there won't be any other culture that could push humanity forward. People tend to become complacent if they aren't feeling competition, leading to a downfall, and that is true for all aspects of life. It's like we are all in some sort of Reinforcement Learning algorithm and unifying the language portion would significantly restrict the exploration phase, the one that moves world forward. There are many concepts unique to each language, different emphasis on different things, different emotional response to different sentences, leading to unique ways of exploring the universe, and that all would be lost.
- azangru 6y ago
  > unifying the mindsets of all people via the same language There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language. To me, the prospect of cultures having to communicate with each other via such a broken mechanism as translation is far more repugnant than a linguistically unified superculture within which ideas flow unimpeded. The more language users there are, the more capable of expression the language will become; so no, I am not particularly worried about getting stuck in a local maximum.
  - djedr 6y ago
    I think it goes both ways: language shapes mind and mind shapes language. It's a feedback loop.

LockAndLol 6y ago

Though i agree with the main point, the blog post went off on a tangent about how some other languages are in danger because of English. IMO if the speakers of the language do not: 1. Provide materials in their own language 2. Consume materials in their own language 3. Actively and exclusively contribute to the world in English it is already a statement about the necessity of their language: it isn't. Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? If its own speakers kill the language (actively or passively) who are we to say they shouldn't?

slx26 6y ago
> Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? Because rational choices at an individual level do not necessarily align with rational choices at a social collective level. "Meditations on Moloch" is a good read on this topic. You can easily see how collectively the language diversity is positive, but individually it kinda makes sense to bet for the biggest language instead. And to be fair, there's a lot of people investing in their own languages, just not enough, or not with enough power.
yorwba 6y ago
I assume you're referring to this paragraph: A continuing lack of technological inclusion will not only exacerbate the language divide but it may also drive speakers of unsupported languages and dialects to high-resource languages with better technological support, further endangering such language varieties. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, to lower language and literacy barriers, we need to apply our models to non-English languages. Have you considered what this means, concretely, for a speaker of an endangered language? Suppose I want to "invest" into my language and 1. Provide materials: - Written? I need some kind of keyboard support. In the worst case, this might entail getting new characters added to Unicode. Or I could just use paper, but then I lose out on all the benefits of digital text editing. - Spoken/Signed? In some ways, that's easier, since I can just record myself. But if I want closed captions or some kind of transcript, I can't make use of automatic tools, I have to transcribe it completely manually. 2. Consume materials: - But how do I find them? What if searching for words in my language mostly turns up results in other languages because the search engine assumes I misspelled my query? - It might happen that the text I'm trying to read uses characters that were considered "too rare" to be included in the default font on my device. On most versions of Android, it's not possible to install a new font to fix that. I'm not sure about other mobile operating systems. Those are problems that don't require speakers of the language to do something differently (such as just using English instead...) but instead need technical solutions that can very well be implemented by someone who doesn't speak the language. I don't speak an endangered language myself, but I've written code to support dozens of languages I don't speak, because someone who did speak the language asked for a particular change that would make life easier for them.

627467 6y ago

I'm reading lots of comments reacting against the premise of the article in name of some kind of efficiency or inherent value of a universal standar. I wonder if commenters have considered that Research is useful to understand the present and more importantly prepare for the future. While English being useful língua Franca today seems to back decision to just focus on English research the point is: will this English uptake continue at same rate in future? Isn't there some kind of bubble effect where here in hn many think that (as non English native speakers) we can all engage so we think English is the only useful thing for us to understand? What about all possible unknown Mandarin NLP research (which I think many agree) may be useful for non-mandarin speakers? Also, isn't one of the points of Information Technology to enable/create value in a long tail of experiences? Focusing on a single attribute of a substantial part of the world seems counter productive and shortsighted...

tanilama 6y ago

With new direction of research, like pre-taining and subword vocab, the models now are mostly language agnostic. English or not, it is just sequence of tokens. Difference mainly lies in data abundance, where the distribution is severly skewed.

mindvirus 6y ago

Can anyone chime in to how Chinese NLP compares? I speak Mandarin as a second language at an intermediate level, and from my perspective it has some interesting properties compared to English (my native language) that seem like they might make NLP easier in ways that English NLP might be hard: - The grammar for standard Mandarin is much simpler than English. No verb conjugations or strange tense rules ("go" -> "went")! - Tense and even voice are explicit and additive - i.e. adding "呢" or "吧" to a sentence. - Written, it seems less ambiguous. i.e. "duck" or "grave" have different meanings in English based on context, but that seems much much rarer in Chinese - two words spoken the same way but with different meanings usually have different written forms.

Tainnor 6y ago
Mandarin has even less morphology than English[1], so to some extent, working on Mandarin would not "fix" some of the issues of NLP models often performing poorly on morphology-rich languages. But it's possible that focusing more on Mandarin would still lead to improvements in other areas, because the two languages are otherwise quite different (I wonder how well speech recognition works for tonal languages, for example?). [1] Possibly not without reason: there is some indication that languages spoken by a large number of speakers tend to become structurally simpler, while very small isolated languages can often develop surprising complexity.

bfung 6y ago

A cynical take supporting the above: NLP in other languages can help sell better ads in those languages. You all can follow that chain of thought down to it's corporate conclusion =)