3 ms·

I still wonder why we're training models on all these languages especially when they have different alphabets etc, we've got solid translators, wouldn't it be m

I still wonder why we're training models on all these languages especially when they have different alphabets etc, we've got solid translators, wouldn't it be more parameter dense to target one language for all data and tokens, and then have a layer specifically for input and output translation?

eightysixfour 2y ago
I'd be interested to know if adding more languages makes them more or less performant. It is my understanding that you have to add code for the models to perform well, for example.
- cchance 2y ago
  My issue and questions isn't so much languages but character sets, expanding say an ascii dataset to include all the non-ascii language glyphs or other unicode characters has got to balloon the parameter count
ijustlovemath 2y ago
more languages gives deeper semantic understanding; I think it only helps with diversity of data, which ultimately improves outputs
sva_ 2y ago
You'll have a lower bound on the quality of the translator you're using. There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.
- econ 2y ago
  Each language has unique tools. If you have a word for something or even better a whole set of words the conversation works a lot better than in a language that has nothing of the kind. English elaborately talks about all kinds of Communities. Dutch does have a word like it but it is almost never used. Or, how should we talk about the kind of snow if we have only one word? https://watchingtheswedes.com/2018/02/28/50-words-for-snow/ https://watchingtheswedes.com/2018/02/28/50-words-for-snow/
cma 2y ago
More languages helps it at novel translation tasks, models have been tested with languages not in/barely in the corpus and a translation book in context and were able to do an ok job. You'll also have things like mulimodal where you want to preserve all the tonality and emphasis in the input language.
astrange 2y ago
Bitter lesson says any kind of specialization is not worth it[0]. Also, you want to be able to have mixed language conversations, like defining a Chinese word in English. [0] but it might be worth it if you need a smaller model because then there are tradeoffs again.

eightysixfour 2y ago

I'd be interested to know if adding more languages makes them more or less performant. It is my understanding that you have to add code for the models to perform well, for example.

cchance 2y ago
My issue and questions isn't so much languages but character sets, expanding say an ascii dataset to include all the non-ascii language glyphs or other unicode characters has got to balloon the parameter count

ijustlovemath 2y ago

more languages gives deeper semantic understanding; I think it only helps with diversity of data, which ultimately improves outputs

sva_ 2y ago

You'll have a lower bound on the quality of the translator you're using. There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.

econ 2y ago
Each language has unique tools. If you have a word for something or even better a whole set of words the conversation works a lot better than in a language that has nothing of the kind. English elaborately talks about all kinds of Communities. Dutch does have a word like it but it is almost never used. Or, how should we talk about the kind of snow if we have only one word? https://watchingtheswedes.com/2018/02/28/50-words-for-snow/ https://watchingtheswedes.com/2018/02/28/50-words-for-snow/

cma 2y ago

More languages helps it at novel translation tasks, models have been tested with languages not in/barely in the corpus and a translation book in context and were able to do an ok job. You'll also have things like mulimodal where you want to preserve all the tonality and emphasis in the input language.

astrange 2y ago

Bitter lesson says any kind of specialization is not worth it[0]. Also, you want to be able to have mixed language conversations, like defining a Chinese word in English. [0] but it might be worth it if you need a smaller model because then there are tradeoffs again.