3 ms·

There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of

There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of languages and can benefit from a larger corpus of various different languages. It’s like Chomsky, but with the Norvig approach.

fenomas 4y ago
Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?
- e12e 4y ago
  That's probably what op meant by "it's like Chomsky" :)
- wongarsu 4y ago
  It probably goes even beyond that. Most languages are related to other languages, and somebody good in German and Norwegian could do an ok job transcribing Swedish with minimal training. Similarly, learning English is a lot easier for German speakers than for Chinese, even though German and English are quite different in pronunciation. The more languages from a language family a human speaks, the easier it is for them to learn each additional language from that family. It stands to reason that deep learning can benefit similarly.
hackernewds 4y ago
> It's like Chomsky, but with the Norvig approach. what does this mean?
- ot 4y ago
  Chomsky postulates that all languages boil down to a universal set of rules, and he went as far as saying that this is biologically motivated, as if there was a "language device" in the brain. Norvig was one of the pioneers in proposing that NLP could be solved by statistical machine learning on large quantities of data. So the joke here is that we're using large-scale machine learning to infer the underlying universal structure of language. --- EDIT: After writing the comment above I asked ChatGPT-4 the same question, and it came up with the following answer, which I think is flawless. --- The last sentence in the paragraph is referencing two well-known figures in the field of linguistics and artificial intelligence: Noam Chomsky and Peter Norvig. Noam Chomsky is a linguist who proposed the idea of a "universal grammar," suggesting that all human languages share a common underlying structure, and that humans have an innate ability to acquire language. This idea implies that even with limited exposure to a specific language, humans can still learn it thanks to this innate structure. Peter Norvig is a computer scientist and artificial intelligence researcher who is known for advocating a more data-driven, statistical approach to natural language processing. This approach emphasizes the importance of large datasets in teaching machines to understand and process language, rather than relying on pre-defined rules or structures. The sentence "It's like Chomsky, but with the Norvig approach" suggests that the described language model incorporates elements of both Chomsky's universal grammar and Norvig's data-driven approach. The model benefits from an underlying structure (akin to Chomsky's universal grammar) while also leveraging a large corpus of various languages (as advocated by Norvig) to improve its performance in understanding and processing language.
  - oriettaxx 4y ago
    wow!

fenomas 4y ago

Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?

e12e 4y ago
That's probably what op meant by "it's like Chomsky" :)
wongarsu 4y ago
It probably goes even beyond that. Most languages are related to other languages, and somebody good in German and Norwegian could do an ok job transcribing Swedish with minimal training. Similarly, learning English is a lot easier for German speakers than for Chinese, even though German and English are quite different in pronunciation. The more languages from a language family a human speaks, the easier it is for them to learn each additional language from that family. It stands to reason that deep learning can benefit similarly.

hackernewds 4y ago

> It's like Chomsky, but with the Norvig approach. what does this mean?

ot 4y ago
Chomsky postulates that all languages boil down to a universal set of rules, and he went as far as saying that this is biologically motivated, as if there was a "language device" in the brain. Norvig was one of the pioneers in proposing that NLP could be solved by statistical machine learning on large quantities of data. So the joke here is that we're using large-scale machine learning to infer the underlying universal structure of language. --- EDIT: After writing the comment above I asked ChatGPT-4 the same question, and it came up with the following answer, which I think is flawless. --- The last sentence in the paragraph is referencing two well-known figures in the field of linguistics and artificial intelligence: Noam Chomsky and Peter Norvig. Noam Chomsky is a linguist who proposed the idea of a "universal grammar," suggesting that all human languages share a common underlying structure, and that humans have an innate ability to acquire language. This idea implies that even with limited exposure to a specific language, humans can still learn it thanks to this innate structure. Peter Norvig is a computer scientist and artificial intelligence researcher who is known for advocating a more data-driven, statistical approach to natural language processing. This approach emphasizes the importance of large datasets in teaching machines to understand and process language, rather than relying on pre-defined rules or structures. The sentence "It's like Chomsky, but with the Norvig approach" suggests that the described language model incorporates elements of both Chomsky's universal grammar and Norvig's data-driven approach. The model benefits from an underlying structure (akin to Chomsky's universal grammar) while also leveraging a large corpus of various languages (as advocated by Norvig) to improve its performance in understanding and processing language.
- oriettaxx 4y ago
  wow!