2 ms·

I think it's pretty obvious how the languages were selected: they used all languages where data was readily available. And I think that's a pretty good choice,

by yorwba 6y ago

I think it's pretty obvious how the languages were selected: they used all languages where data was readily available. And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages when the goal is to test generalisation across languages. (Okay, limited computing resources would be a reason, but Google can afford it.) I agree that there should be some discussion on why transfer to certain languages is easier than to others (and maybe that's actually in the paper, I've only read the blog post so far). Language relatedness is one obvious explanation, but another possibility is contamination of the training data. It's rare to find Cantonese on the internet marked with lang="yue", much more often it's simply labeled as Chinese, i.e. lang="zh". That makes it hard to collect "pure" Standard Mandarin data that doesn't include other varieties.

tasogare 6y ago
> And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages There is a reason, which is akin to why balanced corpus are used for capturing various aspects of a language: language features are unevenly distributed between languages but quite consistent within a family. Let's take the example of word order [1]. The SVO order is very common on the raw number of languages, but actually represented by a small number of families (4 times less families than SOV for about the same number of languages). Which mean if the so-called "agnostic" model is trained mainly on languages of the same family (spoiler: it is), the raw number of languages used can be high, yet the difference between them is minor so the task is made easier. In ML term the model is kind of overfitting the feature from the language that are the most represented, yet the result can pass as good when tested on languages similar to those used for learning. This is clearly what could be happening here, from the previous example I gave. Also Catalan: 75% accuracy for a language that is linguitically very close to French and Spanish (themselves close together) which are the top 5th and 9th language in number of data in the training set. The first two best non constructed languages are the closest to the 4th language in the training set... Besides, I don't think anyone really on lang tag as it often missing or simply wrong. Besides, taking Cantonese from Standard Chinese is a very easy task as their is a few words that appears very frequently in Cantonese and never in SM (係, 嘅, 冇 for instance). [1] https://en.wikipedia.org/wiki/Word_order#Distribution_of_word_order_types https://en.wikipedia.org/wiki/Word_order#Distribution_of_wor...

by yorwba 6y ago

tasogare 6y ago

> And I think that's a pretty good choice, since there's no reason to arbitrarily exclude languages There is a reason, which is akin to why balanced corpus are used for capturing various aspects of a language: language features are unevenly distributed between languages but quite consistent within a family. Let's take the example of word order [1]. The SVO order is very common on the raw number of languages, but actually represented by a small number of families (4 times less families than SOV for about the same number of languages). Which mean if the so-called "agnostic" model is trained mainly on languages of the same family (spoiler: it is), the raw number of languages used can be high, yet the difference between them is minor so the task is made easier. In ML term the model is kind of overfitting the feature from the language that are the most represented, yet the result can pass as good when tested on languages similar to those used for learning. This is clearly what could be happening here, from the previous example I gave. Also Catalan: 75% accuracy for a language that is linguitically very close to French and Spanish (themselves close together) which are the top 5th and 9th language in number of data in the training set. The first two best non constructed languages are the closest to the 4th language in the training set... Besides, I don't think anyone really on lang tag as it often missing or simply wrong. Besides, taking Cantonese from Standard Chinese is a very easy task as their is a few words that appears very frequently in Cantonese and never in SM (係, 嘅, 冇 for instance). [1] https://en.wikipedia.org/wiki/Word_order#Distribution_of_word_order_types https://en.wikipedia.org/wiki/Word_order#Distribution_of_wor...