3 ms·
> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advance
by marcinzm 6y ago
> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020).
Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down.
> Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
- thomasahle 6y ago> I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from. I mostly agree with you, but it does seem like current models require more data to learn language than baby humans. This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
- slx26 6y agoBut baby humans tend to associate phonetic symbols (spoken words) to what they are seeing, and the learning method is very different. Their context is much broader, helped by multiple senses. NLP is extremely contextless compared to what humans do. You can't talk about "more data" so easily.
- Eridrus 6y agoOne random note I heard on a podcast about child development is that children do not improve their language skills by watching tv or educational material and that they need feedback to learn. Contrast that with language models which do nothing besides watching what people are writing, it potentially points to a reason who the amount of data is not comparable.
- rvense 6y agoEven more crucial is that children learn language in situations of actual language use. The things and situations that are spoken about are often at hand in some way, and there is a large social element. No matter how close you might get to "the language learning algorithm", if all you're feeding it is text, it's not going to learn the same thing that kids learn. The data is simply not there.