3 ms·
> We demonstrate that utilizing a large unlabeled multilingual dataset to pre-train the encoder of our model and fine-tuning on a smaller set of labeled data en
by RC_ITR 4y ago
> We demonstrate that utilizing a large unlabeled multilingual dataset to pre-train the encoder of our model and fine-tuning on a smaller set of labeled data enables us to recognize these under-represented languages. Moreover, our model training process is effective for adapting to new languages and data.
What this implies is that the core encoder model (trained on popular languages with a ton of data) does a really good job at learning the generalized basics of any language (period).
Then, they enter a comparatively minuscule amount of labeled data from the small languages at the end (supervised fine tuning - since parameters are tuned iteratively, the latest training data has the biggest impact on the final performance). The model has enough of a general “understanding” of grammar across languages that it can fill in the gaps it doesn’t get from that small labeled set.
Thats the true crazy part. The pre trained encoder is an ersatz Rosetta Stone.
- int_19h 4y agoDoesn't this also strongly imply that universal grammar is indeed a thing?
- RC_ITR 4y agoWell, it’s certainly not evidence against, but it’s still possible that there are n-families of unrelated grammar structures and Google’s training caught enough examples of each of those families via popular languages. Also, for all we know, they failed on a bunch of languages (unless there’s a paper somewhere I missed, but sadly the days of good AI papers from for-profits seems to be over).
- numpad0 4y agoCan it be more like a universal grammar coprocessor, with finite complexity?
- PeterisP 4y agoNot necessarily, the concept of universal grammar is more about the concept of innate grammar that has to be universal for any homo sapiens language, but this empirical observation is more about the similarity of particular languages, many of which are significantly related to each other. For example, if it had historically turned out (or will turn out in some future) that everyone on Earth speaks some variant of English or other Indo-European language; or some variant of Chinese or other Sino-Tibetan, then there would be many features of grammar that technically are universal across the population but that wouldn't be evidence that "universal grammar" (as used by Chomsky) is indeed a thing.
- creamyhorror 4y agoCool though not entirely surprising that a network that learns how to process many human languages might develop internal subnets/structures that generalise to nearly all human languages. Could be seen as a humanlang coprocessor, as a sibling says. Understanding those generic internal structures would be fascinating.
- thelittleone 4y agoIf it can do this, I wonder if it could be adapted to develop a language that is universal and easy to learn.
- deskamess 4y agoRight? I think understanding, or perhaps a better word is representing, the internal structure is also best left to a machine to extract. If it can categorize/cluster into various neural neighborhoods then we can feed that output (or perhaps many outputs from different runs) into another system that is responsible for explaining to humans.