3 ms·
>> Recent models have repeatedly matched human-level performance on increasingly difficult benchmarks—that is, in English using labelled datasets with thousands
by YeGoblynQueenne 6y ago
>> Recent models have repeatedly matched human-level performance on
increasingly difficult benchmarks—that is, in English using labelled datasets
with thousands and unlabelled data with millions of examples. In the process,
as a community we have overfit to the characteristics and conditions of
English-language data. In particular, by focusing on high-resource languages,
we have prioritised methods that work well only when large amounts of labelled
and unlabelled data are available.
This is an interesting observation. Training on large datasets tends to be
framed as a strength, e.g. we have recently seen articles praising OpenAI's
GPT-3 for being, well, big. The truth is that model size (and the assorted
dataset size) is a bug, not a feature. It is the result of a dearth of
research on approaches with low sample complexity. Or in other words, it's the
result of consistently picking the low-hanging fruit and calling sour grapes
on any domain for which there isn't sufficiently large data ("who cares about
Swahilli?").
>> In contrast, most current methods break down when applied to the
data-scarce conditions that are common for most of the world's languages. Even
recent advances in pre-training language models that dramatically reduce the
sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder,
2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of
clean, unlabelled data, which is not available for most of the world's
languages (Artetxe et al., 2020). Doing well with few data is thus an ideal
setting to test the limitations of current models—and evaluation on
low-resource languages constitutes arguably its most impactful real-world
application.
And this is interesting to read in the context of the recent manic hype about
GPT-3, itself described in a paper titled "Language models are few-shot
learners", hilariously attempting to sweep under the carpet the amount of data
and compute required to get to the point where a language model can do "few"
shot learning.
In general, the excitement about the successes that have come from training
large, deep neural nework models with very large datasets has only served to
avoid posing the obvious question about such data-hungry approaches: what do
we do when there isn't a lot of data? Human language with all its minutely
fragmented diversity, turns out to be just that kind of domain.
- ShamelessC 6y agoI see this argument a lot. I have essentially zero ml experience, but isn't it fair to say that DNA indeed carries a similarly large amount of data? Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off. Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.
- YeGoblynQueenne 6y agoHumans seem to come into the world with very good quality biases that allow us to learn any human language but I don't think anyone has ever found DNA encoding information about, say, the English language. An I misunderstanding your comment? If so, I apologise- I'm not sure where DNA comes in. >> Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off. This is a complicated matter and as I'm not a linguist I don't know what theories there even exist. However, I think a human child is perfectly capable of inventing an entire language. Small groups of children are certainly capable of doing that in the early stages of their lives. For example, see Creole languages [1] which are the languages invented by the children of people who only have a pidgin as a common language. A pidgin is not a fully developed language and pidgins are used e.g. for communication between migrant communities with different origins and who do not have a common language, while a Creole is a fully developed natural language that is invented by the migrants' children, who have their parents' share pidgin as a maternal language. The children start with their parents' pidgin and develop it into a fully formed natural language with all the bells and whistles. It's quite a thing to read about really, have a look at the wikipedia article I link above it's very interesting. Also, as far as I understand it there's a lot that we don't know about how the first human languages were created. From some short readings, the current understanding is that at some point, humans must have become capable of producing natural language whereas before they weren't. The way things usually work the turning point for this was most likely some random mutation that affected a very few individuals, possibly no more than two. So the first natural language was probably invented not by an entire society but by a couple of people, probably a pair of siblings. The idea is that the ability to use natural language gave those very few individuals an important advantage that ensured its propagation through generations until our time. In any case that all goes to invention of a new language. Learning an existing language is quite another matter. Children who grow up to learn their maternal language don't have access to "millions of years of data". They certainly don't have access to a few billion examples of utterances in their maternal language, like the datasets that language models are trained on. That is to say, kids learn whatever language is their maternal language very, very quickly as they grow up and from very few examples. And they're capable of learning any of the few thousand natural languages, beyond English. For me at least this lends credence to an idea of an innate bias to learn a certain type of language, a type broad enough to cover every human language. To put a name on this idea, that's through and through Noam Chomsky's idea of an innate "Universal Grammar". See wikipedia about criticisms of UG and linguistic nativism in general [2]. ____________ [1] https://en.wikipedia.org/wiki/Creole_language https://en.wikipedia.org/wiki/Creole_language [2] https://en.wikipedia.org/wiki/Universal_grammar#Criticisms https://en.wikipedia.org/wiki/Universal_grammar#Criticisms