5 ms·

They used data from youtube, with multiple selection methods. A little supervised data for 72 languages (90k hours of audio with text labels provided), some pse

by est31 4y ago

They used data from youtube, with multiple selection methods. A little supervised data for 72 languages (90k hours of audio with text labels provided), some pseudo-supervised data in english (100k hours; the english labels were generated by having a model do the labeling), and a LOT of unsupervised data in 568 languages (12.1 million hours). That last group has no labels (aka captions) available, just the audio, but they create pseudo-labels by using the FLEURS dataset. I'm not really sure how this works as FLEURS itself only has 102 languages... I guess in different words, it just makes some phonetic impression of how that language should be written and then compares itself to how well it can do that impression?

williamcotton 4y ago
There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of languages and can benefit from a larger corpus of various different languages. It’s like Chomsky, but with the Norvig approach.
- fenomas 4y ago
  Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?
  - e12e 4y ago
    That's probably what op meant by "it's like Chomsky" :)
  - wongarsu 4y ago
    It probably goes even beyond that. Most languages are related to other languages, and somebody good in German and Norwegian could do an ok job transcribing Swedish with minimal training. Similarly, learning English is a lot easier for German speakers than for Chinese, even though German and English are quite different in pronunciation. The more languages from a language family a human speaks, the easier it is for them to learn each additional language from that family. It stands to reason that deep learning can benefit similarly.
- hackernewds 4y ago
  > It's like Chomsky, but with the Norvig approach. what does this mean?
  - ot 4y ago
    Chomsky postulates that all languages boil down to a universal set of rules, and he went as far as saying that this is biologically motivated, as if there was a "language device" in the brain. Norvig was one of the pioneers in proposing that NLP could be solved by statistical machine learning on large quantities of data. So the joke here is that we're using large-scale machine learning to infer the underlying universal structure of language. --- EDIT: After writing the comment above I asked ChatGPT-4 the same question, and it came up with the following answer, which I think is flawless. --- The last sentence in the paragraph is referencing two well-known figures in the field of linguistics and artificial intelligence: Noam Chomsky and Peter Norvig. Noam Chomsky is a linguist who proposed the idea of a "universal grammar," suggesting that all human languages share a common underlying structure, and that humans have an innate ability to acquire language. This idea implies that even with limited exposure to a specific language, humans can still learn it thanks to this innate structure. Peter Norvig is a computer scientist and artificial intelligence researcher who is known for advocating a more data-driven, statistical approach to natural language processing. This approach emphasizes the importance of large datasets in teaching machines to understand and process language, rather than relying on pre-defined rules or structures. The sentence "It's like Chomsky, but with the Norvig approach" suggests that the described language model incorporates elements of both Chomsky's universal grammar and Norvig's data-driven approach. The model benefits from an underlying structure (akin to Chomsky's universal grammar) while also leveraging a large corpus of various languages (as advocated by Norvig) to improve its performance in understanding and processing language.
    - oriettaxx 4y ago
      wow!
RC_ITR 4y ago
> We demonstrate that utilizing a large unlabeled multilingual dataset to pre-train the encoder of our model and fine-tuning on a smaller set of labeled data enables us to recognize these under-represented languages. Moreover, our model training process is effective for adapting to new languages and data. What this implies is that the core encoder model (trained on popular languages with a ton of data) does a really good job at learning the generalized basics of any language (period). Then, they enter a comparatively minuscule amount of labeled data from the small languages at the end (supervised fine tuning - since parameters are tuned iteratively, the latest training data has the biggest impact on the final performance). The model has enough of a general “understanding” of grammar across languages that it can fill in the gaps it doesn’t get from that small labeled set. Thats the true crazy part. The pre trained encoder is an ersatz Rosetta Stone.
- int_19h 4y ago
  Doesn't this also strongly imply that universal grammar is indeed a thing?
  - RC_ITR 4y ago
    Well, it’s certainly not evidence against, but it’s still possible that there are n-families of unrelated grammar structures and Google’s training caught enough examples of each of those families via popular languages. Also, for all we know, they failed on a bunch of languages (unless there’s a paper somewhere I missed, but sadly the days of good AI papers from for-profits seems to be over).
  - numpad0 4y ago
    Can it be more like a universal grammar coprocessor, with finite complexity?
  - PeterisP 4y ago
    Not necessarily, the concept of universal grammar is more about the concept of innate grammar that has to be universal for any homo sapiens language, but this empirical observation is more about the similarity of particular languages, many of which are significantly related to each other. For example, if it had historically turned out (or will turn out in some future) that everyone on Earth speaks some variant of English or other Indo-European language; or some variant of Chinese or other Sino-Tibetan, then there would be many features of grammar that technically are universal across the population but that wouldn't be evidence that "universal grammar" (as used by Chomsky) is indeed a thing.

by est31 4y ago

williamcotton 4y ago

There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of languages and can benefit from a larger corpus of various different languages. It’s like Chomsky, but with the Norvig approach.

fenomas 4y ago
Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?
- e12e 4y ago
  That's probably what op meant by "it's like Chomsky" :)
- wongarsu 4y ago
  It probably goes even beyond that. Most languages are related to other languages, and somebody good in German and Norwegian could do an ok job transcribing Swedish with minimal training. Similarly, learning English is a lot easier for German speakers than for Chinese, even though German and English are quite different in pronunciation. The more languages from a language family a human speaks, the easier it is for them to learn each additional language from that family. It stands to reason that deep learning can benefit similarly.
hackernewds 4y ago
> It's like Chomsky, but with the Norvig approach. what does this mean?
- ot 4y ago
  Chomsky postulates that all languages boil down to a universal set of rules, and he went as far as saying that this is biologically motivated, as if there was a "language device" in the brain. Norvig was one of the pioneers in proposing that NLP could be solved by statistical machine learning on large quantities of data. So the joke here is that we're using large-scale machine learning to infer the underlying universal structure of language. --- EDIT: After writing the comment above I asked ChatGPT-4 the same question, and it came up with the following answer, which I think is flawless. --- The last sentence in the paragraph is referencing two well-known figures in the field of linguistics and artificial intelligence: Noam Chomsky and Peter Norvig. Noam Chomsky is a linguist who proposed the idea of a "universal grammar," suggesting that all human languages share a common underlying structure, and that humans have an innate ability to acquire language. This idea implies that even with limited exposure to a specific language, humans can still learn it thanks to this innate structure. Peter Norvig is a computer scientist and artificial intelligence researcher who is known for advocating a more data-driven, statistical approach to natural language processing. This approach emphasizes the importance of large datasets in teaching machines to understand and process language, rather than relying on pre-defined rules or structures. The sentence "It's like Chomsky, but with the Norvig approach" suggests that the described language model incorporates elements of both Chomsky's universal grammar and Norvig's data-driven approach. The model benefits from an underlying structure (akin to Chomsky's universal grammar) while also leveraging a large corpus of various languages (as advocated by Norvig) to improve its performance in understanding and processing language.
  - oriettaxx 4y ago
    wow!

RC_ITR 4y ago

> We demonstrate that utilizing a large unlabeled multilingual dataset to pre-train the encoder of our model and fine-tuning on a smaller set of labeled data enables us to recognize these under-represented languages. Moreover, our model training process is effective for adapting to new languages and data. What this implies is that the core encoder model (trained on popular languages with a ton of data) does a really good job at learning the generalized basics of any language (period). Then, they enter a comparatively minuscule amount of labeled data from the small languages at the end (supervised fine tuning - since parameters are tuned iteratively, the latest training data has the biggest impact on the final performance). The model has enough of a general “understanding” of grammar across languages that it can fill in the gaps it doesn’t get from that small labeled set. Thats the true crazy part. The pre trained encoder is an ersatz Rosetta Stone.

int_19h 4y ago
Doesn't this also strongly imply that universal grammar is indeed a thing?
- RC_ITR 4y ago
  Well, it’s certainly not evidence against, but it’s still possible that there are n-families of unrelated grammar structures and Google’s training caught enough examples of each of those families via popular languages. Also, for all we know, they failed on a bunch of languages (unless there’s a paper somewhere I missed, but sadly the days of good AI papers from for-profits seems to be over).
- numpad0 4y ago
  Can it be more like a universal grammar coprocessor, with finite complexity?
- PeterisP 4y ago
  Not necessarily, the concept of universal grammar is more about the concept of innate grammar that has to be universal for any homo sapiens language, but this empirical observation is more about the similarity of particular languages, many of which are significantly related to each other. For example, if it had historically turned out (or will turn out in some future) that everyone on Earth speaks some variant of English or other Indo-European language; or some variant of Chinese or other Sino-Tibetan, then there would be many features of grammar that technically are universal across the population but that wouldn't be evidence that "universal grammar" (as used by Chomsky) is indeed a thing.