4 ms·

Typical ML-based paper where everything is about machine learning and nothing is about languages... The biggest issue is nothing is said about how the languages

Typical ML-based paper where everything is about machine learning and nothing is about languages... The biggest issue is nothing is said about how the languages were selected. That’s the basis when dealing with multilingual languages as the features they share or not is important and will impact the results. In this respect the last figure is the more interesting one: it mentions ido and interlingua, two constructed languages. Why does they get included since NLP is supposed to deal with natural languages? It not a bad choice per se, but the selection criteria needs to be explain more than just picking languages to get a large number on a research paper. There is also no reflection on why some languages got better results than others. Again looking at the last figure in the top three best performing languages with no data two are Sinitic languages (Cantonese and Wu) which likely happens since their closeness to standard Chinese for which there is huge ressources available for training. On the reverse Breton for which there is probably very few apparented language data in the initial models and training set besides Irish get very poor results, which tends to show the model actually don’t transfer very well if at all...

gleenn 6y ago
In one of the graphs, the x-axis has the language code for the languages they used I think: https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAGZw/_qFDiNyDi7cMeIBtUa-qoTJ9_6ODKtLtwCLcBGAsYHQ/s1621/image1.png https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAG...
mkasu 6y ago
NLP is not my main field but still relevant to my work because I often use models and resources from NLP as tools. I'm also personally interested in Linguistics and Languages so I follow related news, sometimes attend NLP conferences and follow people in those fields on Social Media. It is very concerning how few thought is usually put into linguistic or language characteristics when dealing with these topics. I also rarely see cultural considerations etc. Basically everything is considered as "machine learning will hopefully get this right if having enough data" which is unfortunate (ML is a great tool but the conferences are about language processing). Another big issue I noticed is that a majority of research only targets or evaluates English texts. In many cases the language is not even specified (although it is clear they use English from figures or examples). I even heard people complaining that work on non-English data is treated as too minor by many reviewers so stuff like that often just gets rejected. I think this is a really weird development for a field which centers around natural languages.
- tasogare 6y ago
  Yes, this is exactly my point. NLP is about processing language (which have a century old field dedicated to it) yet the new trend is to totally discard that as a minor details. It's not. It's also fine if people mostly focus on English but then they should be clear about it and not claim to address language in general when they are in fact doing English processing in particular.
- IfOnlyYouKnew 6y ago
  While I sort-of recognize the emotion you describe in myself, it cannot be ignored that these ignoramuses are simply blowing "traditional" research out of the water in terms of results. That's true across the board, from NLP to image data to computational biology. It's also a bit simplified to consider it a bifurcation between "traditional" linguists and AI experts entirely ignorant of the discipline. Long before the current wave of AI started, Google liked to hire linguists and computational scientists. These teams probably do have plenty of subject matter experts, but for now they are reaping the low-hanging fruits of the suddenly-improved generic methods. As the marginal improvements are inevitably diminished, subject matter will become more salient again. I'm a computational biologist by training, and have great appreciation for the often beautiful algorithms, many created in the 70s or 80s and allowing then-spectacular feats of tackling large datasets. Unfortunately, it isn't always obvious how to transfer that knowledge to the new way of doing things.
  - mkasu 6y ago
    Yes, the seeming performance of (especially) neural models compared to traditional models is probably the main factor. Although, some voices[1] argue that traditional or much simpler approaches still often do a similar job compared to super over-engineered models, especially when going even slightly beyond an existing target-dataset or task. I'd argue, that improving the ML models is really the job of ML researchers and should be mainly targeting ML conferences like AAAI (Adv. of AI). In other conferences (directly targeting NLP, CV, Comp. Biology, etc.) it should be the main job to combine those models with the domain-specific characteristics (e.g., language information for NLP) or "traditional" methods to make it an interesting discussion. I was recently doing reviewing for a multimedia conference and quite a lot of the papers I reviewed were basically pure ML papers. A colleague had the same experience. 1: https://arxiv.org/abs/1907.06902 https://arxiv.org/abs/1907.06902
yorwba 6y ago
I think it's pretty obvious how the languages were selected: they used all languages where data was readily available. And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages when the goal is to test generalisation across languages. (Okay, limited computing resources would be a reason, but Google can afford it.) I agree that there should be some discussion on why transfer to certain languages is easier than to others (and maybe that's actually in the paper, I've only read the blog post so far). Language relatedness is one obvious explanation, but another possibility is contamination of the training data. It's rare to find Cantonese on the internet marked with lang="yue", much more often it's simply labeled as Chinese, i.e. lang="zh". That makes it hard to collect "pure" Standard Mandarin data that doesn't include other varieties.
- tasogare 6y ago
  > And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages There is a reason, which is akin to why balanced corpus are used for capturing various aspects of a language: language features are unevenly distributed between languages but quite consistent within a family. Let's take the example of word order [1]. The SVO order is very common on the raw number of languages, but actually represented by a small number of families (4 times less families than SOV for about the same number of languages). Which mean if the so-called "agnostic" model is trained mainly on languages of the same family (spoiler: it is), the raw number of languages used can be high, yet the difference between them is minor so the task is made easier. In ML term the model is kind of overfitting the feature from the language that are the most represented, yet the result can pass as good when tested on languages similar to those used for learning. This is clearly what could be happening here, from the previous example I gave. Also Catalan: 75% accuracy for a language that is linguitically very close to French and Spanish (themselves close together) which are the top 5th and 9th language in number of data in the training set. The first two best non constructed languages are the closest to the 4th language in the training set... Besides, I don't think anyone really on lang tag as it often missing or simply wrong. Besides, taking Cantonese from Standard Chinese is a very easy task as their is a few words that appears very frequently in Cantonese and never in SM (係, 嘅, 冇 for instance). [1] https://en.wikipedia.org/wiki/Word_order#Distribution_of_word_order_types https://en.wikipedia.org/wiki/Word_order#Distribution_of_wor...

gleenn 6y ago

In one of the graphs, the x-axis has the language code for the languages they used I think: https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAGZw/_qFDiNyDi7cMeIBtUa-qoTJ9_6ODKtLtwCLcBGAsYHQ/s1621/image1.png https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAG...

mkasu 6y ago

NLP is not my main field but still relevant to my work because I often use models and resources from NLP as tools. I'm also personally interested in Linguistics and Languages so I follow related news, sometimes attend NLP conferences and follow people in those fields on Social Media. It is very concerning how few thought is usually put into linguistic or language characteristics when dealing with these topics. I also rarely see cultural considerations etc. Basically everything is considered as "machine learning will hopefully get this right if having enough data" which is unfortunate (ML is a great tool but the conferences are about language processing). Another big issue I noticed is that a majority of research only targets or evaluates English texts. In many cases the language is not even specified (although it is clear they use English from figures or examples). I even heard people complaining that work on non-English data is treated as too minor by many reviewers so stuff like that often just gets rejected. I think this is a really weird development for a field which centers around natural languages.

tasogare 6y ago
Yes, this is exactly my point. NLP is about processing language (which have a century old field dedicated to it) yet the new trend is to totally discard that as a minor details. It's not. It's also fine if people mostly focus on English but then they should be clear about it and not claim to address language in general when they are in fact doing English processing in particular.
IfOnlyYouKnew 6y ago
While I sort-of recognize the emotion you describe in myself, it cannot be ignored that these ignoramuses are simply blowing "traditional" research out of the water in terms of results. That's true across the board, from NLP to image data to computational biology. It's also a bit simplified to consider it a bifurcation between "traditional" linguists and AI experts entirely ignorant of the discipline. Long before the current wave of AI started, Google liked to hire linguists and computational scientists. These teams probably do have plenty of subject matter experts, but for now they are reaping the low-hanging fruits of the suddenly-improved generic methods. As the marginal improvements are inevitably diminished, subject matter will become more salient again. I'm a computational biologist by training, and have great appreciation for the often beautiful algorithms, many created in the 70s or 80s and allowing then-spectacular feats of tackling large datasets. Unfortunately, it isn't always obvious how to transfer that knowledge to the new way of doing things.
- mkasu 6y ago
  Yes, the seeming performance of (especially) neural models compared to traditional models is probably the main factor. Although, some voices[1] argue that traditional or much simpler approaches still often do a similar job compared to super over-engineered models, especially when going even slightly beyond an existing target-dataset or task. I'd argue, that improving the ML models is really the job of ML researchers and should be mainly targeting ML conferences like AAAI (Adv. of AI). In other conferences (directly targeting NLP, CV, Comp. Biology, etc.) it should be the main job to combine those models with the domain-specific characteristics (e.g., language information for NLP) or "traditional" methods to make it an interesting discussion. I was recently doing reviewing for a multimedia conference and quite a lot of the papers I reviewed were basically pure ML papers. A colleague had the same experience. 1: https://arxiv.org/abs/1907.06902 https://arxiv.org/abs/1907.06902

yorwba 6y ago

I think it's pretty obvious how the languages were selected: they used all languages where data was readily available. And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages when the goal is to test generalisation across languages. (Okay, limited computing resources would be a reason, but Google can afford it.) I agree that there should be some discussion on why transfer to certain languages is easier than to others (and maybe that's actually in the paper, I've only read the blog post so far). Language relatedness is one obvious explanation, but another possibility is contamination of the training data. It's rare to find Cantonese on the internet marked with lang="yue", much more often it's simply labeled as Chinese, i.e. lang="zh". That makes it hard to collect "pure" Standard Mandarin data that doesn't include other varieties.

tasogare 6y ago
> And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages There is a reason, which is akin to why balanced corpus are used for capturing various aspects of a language: language features are unevenly distributed between languages but quite consistent within a family. Let's take the example of word order [1]. The SVO order is very common on the raw number of languages, but actually represented by a small number of families (4 times less families than SOV for about the same number of languages). Which mean if the so-called "agnostic" model is trained mainly on languages of the same family (spoiler: it is), the raw number of languages used can be high, yet the difference between them is minor so the task is made easier. In ML term the model is kind of overfitting the feature from the language that are the most represented, yet the result can pass as good when tested on languages similar to those used for learning. This is clearly what could be happening here, from the previous example I gave. Also Catalan: 75% accuracy for a language that is linguitically very close to French and Spanish (themselves close together) which are the top 5th and 9th language in number of data in the training set. The first two best non constructed languages are the closest to the 4th language in the training set... Besides, I don't think anyone really on lang tag as it often missing or simply wrong. Besides, taking Cantonese from Standard Chinese is a very easy task as their is a few words that appears very frequently in Cantonese and never in SM (係, 嘅, 冇 for instance). [1] https://en.wikipedia.org/wiki/Word_order#Distribution_of_word_order_types https://en.wikipedia.org/wiki/Word_order#Distribution_of_wor...