7 ms·

This paper is a good illustration of what I see as a critical shortcoming of modern NLP work: it is all about math and algorithms, there is nothing about actual

This paper is a good illustration of what I see as a critical shortcoming of modern NLP work: it is all about math and algorithms, there is nothing about actual language. The paper has lots of information about: neural network architectures, parameter update equations, learning rules, and inference algorithms. There is nothing about: part of speech categories, relative clauses, morphology, affixes or compound words, the theta criterion, the content/function distinction, verb tenses, agreement, or anything else related to actual linguistic phenomena. To me this seems a bit like the Aristotelian thinkers who tried to reason about physics based on pure mathematical analysis, without any empirical work.

lohankin 10y ago
https://en.wikipedia.org/wiki/Laputa https://en.wikipedia.org/wiki/Laputa
- vog 10y ago
  Care to elaborate?
ktta 10y ago
There is a problem that arises in this case that isn't present in the analogy you gave. The physics they were using pure mathematical analysis was logical. Language, and how humans always change the way they use language is largely illogical. You're right in saying that we can't really get anywhere trying to analyse language piece by piece. But pure mathematical analysis is not getting us great results either.
tlb 10y ago
That's the best part of this style of machine translation: the code knows nothing about parts of speech or other categories that linguists try to force-fit language into. Instead, it learns the structure of both languages from training data (sentence pairs in source and target languages).
Houshalter 10y ago
Because this isn't linguistics work. It's AI work. We are bypassing the impossibly hard problem of natural language processing. Instead we build AIs that can do the hard work for us, and much faster and better than any human linguist ever could.
- fauigerzigerk 10y ago
  Faster yes, but better remains to be seen. I have my doubts on whether machine translation that isn't based on language understanding can ever be trusted not to make egregious mistakes.
  - Houshalter 10y ago
    It's better than previous machine translation systems, I mean. Sure humans can translate better, but they can't describe how in a useful way. Linguists have spent decades trying to come up with hand designed language models, input knowledge about the world in computer readable formats, etc. But this simple AI model can learn to do a better job in a week from raw data.
mafribe 10y ago
Maybe it's your expectation that anything interesting in linguistics needs the language of part of speech categories, relative clauses, morphology ... Maybe that assumption needs to be updated? Maybe the language of network architectures, parameter update equations, learning rules ... is a better way of talking about language? Given the successes that Google's machine translation has had, and the lack of success of traditional linguisitics, it might appear to be the case that this paper is directly about language.
- SomeStupidPoint 10y ago
  It's not a better way of talking about language: it's a better way of talking about how to build a computationally useful model of language with features primarily extracted from data than building language explicitly in is. Pretty much by definition. Google's paper isn't talking about language -- it's talking about how to tune a generalized computational device to build a computationally useful model from a large corpus of data. Of course it sounds different than talking about language, per se.
sidarape 10y ago
> There is nothing about: part of speech categories, relative clauses, morphology, affixes or compound words, the theta criterion, the content/function distinction, verb tenses, agreement, or anything else related to actual linguistic phenomena. When you learned you native tongue, you didn't need to know all of these. You just learned. So, maybe the problem IS about math and algorithms instead of linguistics.
- dmreedy 10y ago
  You certainly didn't need to know the names for them. That doesn't necessarily disqualify them from a role in the learning of language. That said, you still may be right. But at the moment, it's a question of dogma, not empiricism.
  - gradys 10y ago
    Who's to say that neural nets aren't implicitly learning these concepts as they train? In fact, I'd be very surprised if nothing about the networks' internal state corresponded to linguists' models of how language work. Nothing about focusing on the math and not the language means that these concepts don't have a role in how the models learn language. Just as nobody is told about parts of speech as they're acquiring their native tongue, we don't necessarily need to explicitly tell machine learning models about parts of speech in order for them to learn how to use that aspect of language correctly.
- timr 10y ago
  "When you learned you native tongue, you didn't need to know all of these. You just learned." Learning your native language is not at all the same task as translating between languages. People who do translation are usually quite knowledgable in questions of grammar. This is particularly true of people who pick up languages later in life. Even if you know multiple languages "natively", it's often difficult to translate accurately without thinking about grammar. Or, for that matter, to speak/write your own language with any degree of competency -- we study grammar in grade school for a reason.
  - 10y ago
Eridrus 10y ago
We tried solving this by analysing language for decades; the reason we're doing this now is because it actually works a lot better.
- timr 10y ago
  The article directly suggests that isn't the case. There's a (debatable) improvement over Google's existing phrase-based system on these benchmarks, but the whole paper is about the huge effort they've had to put in to get past the shortcomings of the method. A more realistic assessment is that state-of-the-art neural translation is maybe getting as good as phrase-based systems.
  - Houshalter 10y ago
    They claim 60% fewer translation errors than the phrase based system. That's not nothing.
    - Houshalter 10y ago
      Also, see this graph: https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAABPg/-6OhKKPhxT89Vs9HhyKMEnyG_0ncWGjJQCLcB/s1600/image00.png https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAAB... On French > English and English > Spanish it's almost indistinguishable from human translations. On the rest it still gets greater than half the distance between the previous best method and human quality translations. This is a massive step forward.
      - timr 10y ago
        "This is a massive step forward." Take it easy. The graph you're linking to is for a limited survey of 500 isolated sentences, and it's a comparison of mean scores without confidence intervals. The authors themselves admit that it has serious problems as an evaluative tool: "Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work." When you look at the other evaluations in the paper, it's clear that it's about as good as the current best phrase-based tools. In other words, this may be a massive step forward for neural nets, but it puts them at about par for the field.
emcq 10y ago
Christopher Manning, a somewhat famous NLP researcher before and during the deep learning craze, has a great perspective on this that echoes your statements which I recommend reading: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239 http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
SamPhillips 10y ago
This reminds me of the DeepMind bit where the AI beat human players despite not knowing about the edifice of built up knowledge in that domain. I'd expect that an AI could be a great chess player with enough gameplay info and not knowing particular strength scoring, types of famous defenses, etc. In some ways the way that humans have constructed these relationships may actually make learning more difficult. We certainly don't teach 3-year olds about theta criterion, we just speak in 'correct' language and may correct them when they make mistakes.
dmreedy 10y ago
My hunch is that the more apt analogy is to Skinnerian Behaviorists, or apostles of the Hilbert Program. Deep learning is a very a powerful tool, and the current zeal for it has yielded rapid progress in previously stagnating fields. That said, we have yet to see what its limits are in this space; it's still too young to know. It may be that, like in so many cases beforehand, synthesis is the path forward, and there are no silver bullets.
blackkettle 10y ago
Why do think any of those labels are critically important? They are entirely derived from human desire to understand, process, and define a taxonomy for language. There is absolutely no reason to think that they are inherently necessary to achieve the goal or high fidelity translation (or anything else in this domain). Similarly they might be meaningful but inferred naturally through the learning process. If your goal is to map machine learning processes to established dogma, this makes sense; if it is just to build an effective learning machine that solves an extant problem, then it's not.
d_burfoot 10y ago
Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observation: 1) Linguistic knowledge and theory is not useful for NLP work; instead, people should rely entirely on machine learning techniques. 2) Linguistic knowledge is useful in principle, but the field of linguistics has not yet obtained a sufficiently high-quality theory, and it is better to rely on ML than on low-quality theory. Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.
- sushirain 10y ago
  The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition. Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.
- sidarape 10y ago
  > it is better to rely on ML than on low-quality theory. Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.
  - Isamu 10y ago
    In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity. But I agree, it is a not a very explanitory theory since it is too black-box.
- gradys 10y ago
  I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper. It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this. That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML. There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model. [1] - http://arxiv.org/abs/1603.06021 http://arxiv.org/abs/1603.06021
raverbashing 10y ago
I guess there are two issues here: - Most of the available texts have some form of "errors" or are not grammatically complete - Machine translation does not care about grammar, it cares about probabilities NLP with grammar knowledge is feature engineering basically. It is useful when you're "shallow learning" but not so much when you're deep learning
eva1984 10y ago
Linguistics fails to come up with a competitive alternative that leads us to better real application, that is why. In a data driven world, that is nothing to complain about except themselves.
mtrn 10y ago
> There is nothing about: part of speech categories, relative clauses, morphology This is an interesting and deep question. I believe that there are multiple ways to tell the story of a phenomenon. Using POS categories or verb tenses helps us to organize and learn about the medium. It is important for humans to know what a verb tense - in theory and practice - is, to be able to form not just grammatical sentences, but also ones, that express things clearer. However, there is no rule that says that these concepts must be learned first before you can do anything else. In a sense, I see the (neural) machines taking a step back. Whereas earlier attempts at language modeling took into account a lot of linguistic concepts, maybe even tailored to a specific human language, the more end-to-end approaches ignore these things and get better results. What can this tell you about the concepts? They are certainly useful and have been established by a scientific community. The machine, as it reads not just tens or hundreds of sentences but billions, creates a mathematical model, that happens to have aspects, that have very useful real world sides to them. A machine that learns a language model - or that learns translation, can teach us new things about words, and ultimately also about the world. I believe this is a beautiful aspect of the current neural renaissance. ---- Kanerva in a preface to Geometry and Meaning writes: > We think of some sciences, by their very nature, as being more mathematical than others. We even refer to physics and other heavily mathematical sciences as "hard sciences," as if to imply that social sciences and humanities are somehow soft or easy. Another way to see this picture, however, is that mathematics itself is hard and that it would be applied first to sciences that themselves are relatively easy. [...] It matters greatly that the mathematics be appropriate. [...] Mathematics is a tool for coping with complexity. [...] This brings us to the substance of [this] book: the exploration of mathematics that would be appropriate for describing concepts and meaning. The branch of mathematics that has traditionally been associated with meaning is logic. Logic is also discussed here, yet the main emphasis is on identifying mathematical spaces that would lend to modeling of meaning. (http://www.puttypeg.net/book/chapters/foreword.html http://www.puttypeg.net/book/chapters/foreword.html)
arikrak 10y ago
Actually it seems a little like the opposite. Their software is actually translating language quite well in the real empirical world, while the the programs built with language rules were never able to do nearly as well.
joseraul 10y ago
Peter Norvig explains why. http://norvig.com/chomsky.html http://norvig.com/chomsky.html
visarga 10y ago
> There is nothing about: part of speech categories, relative clauses, ... Because it's not the best level of abstraction. Human generated features don't capture all the nuances. In vision there used to be heavy engineering and math involved in feature creation. Now it's all neural nets end to end. Same thing happened to language.
nl 10y ago
Google's recent language modelling paper achieves state of the art performance on exactly these tasks, and it's based on a somewhat vaguely similar architecture. It's becoming pretty clear that many of these concepts may not be fundamental 'things' but just things that make it easy to talk about.
AstralStorm 10y ago
No seriously, this is exactly empirical work. It works somewhat at translating texts. On the other hand, attempts that tried to use linguistic categories failed. A nice new development is multilevel deep networks with explanatory states built in. It will tell you why it detected the thing it did. It may of course be lying or wrong. :)

lohankin 10y ago

https://en.wikipedia.org/wiki/Laputa https://en.wikipedia.org/wiki/Laputa

vog 10y ago
Care to elaborate?

ktta 10y ago

There is a problem that arises in this case that isn't present in the analogy you gave. The physics they were using pure mathematical analysis was logical. Language, and how humans always change the way they use language is largely illogical. You're right in saying that we can't really get anywhere trying to analyse language piece by piece. But pure mathematical analysis is not getting us great results either.

tlb 10y ago

That's the best part of this style of machine translation: the code knows nothing about parts of speech or other categories that linguists try to force-fit language into. Instead, it learns the structure of both languages from training data (sentence pairs in source and target languages).

Houshalter 10y ago

Because this isn't linguistics work. It's AI work. We are bypassing the impossibly hard problem of natural language processing. Instead we build AIs that can do the hard work for us, and much faster and better than any human linguist ever could.

fauigerzigerk 10y ago
Faster yes, but better remains to be seen. I have my doubts on whether machine translation that isn't based on language understanding can ever be trusted not to make egregious mistakes.
- Houshalter 10y ago
  It's better than previous machine translation systems, I mean. Sure humans can translate better, but they can't describe how in a useful way. Linguists have spent decades trying to come up with hand designed language models, input knowledge about the world in computer readable formats, etc. But this simple AI model can learn to do a better job in a week from raw data.

mafribe 10y ago

Maybe it's your expectation that anything interesting in linguistics needs the language of part of speech categories, relative clauses, morphology ... Maybe that assumption needs to be updated? Maybe the language of network architectures, parameter update equations, learning rules ... is a better way of talking about language? Given the successes that Google's machine translation has had, and the lack of success of traditional linguisitics, it might appear to be the case that this paper is directly about language.

SomeStupidPoint 10y ago
It's not a better way of talking about language: it's a better way of talking about how to build a computationally useful model of language with features primarily extracted from data than building language explicitly in is. Pretty much by definition. Google's paper isn't talking about language -- it's talking about how to tune a generalized computational device to build a computationally useful model from a large corpus of data. Of course it sounds different than talking about language, per se.

sidarape 10y ago

> There is nothing about: part of speech categories, relative clauses, morphology, affixes or compound words, the theta criterion, the content/function distinction, verb tenses, agreement, or anything else related to actual linguistic phenomena. When you learned you native tongue, you didn't need to know all of these. You just learned. So, maybe the problem IS about math and algorithms instead of linguistics.

dmreedy 10y ago
You certainly didn't need to know the names for them. That doesn't necessarily disqualify them from a role in the learning of language. That said, you still may be right. But at the moment, it's a question of dogma, not empiricism.
- gradys 10y ago
  Who's to say that neural nets aren't implicitly learning these concepts as they train? In fact, I'd be very surprised if nothing about the networks' internal state corresponded to linguists' models of how language work. Nothing about focusing on the math and not the language means that these concepts don't have a role in how the models learn language. Just as nobody is told about parts of speech as they're acquiring their native tongue, we don't necessarily need to explicitly tell machine learning models about parts of speech in order for them to learn how to use that aspect of language correctly.
timr 10y ago
"When you learned you native tongue, you didn't need to know all of these. You just learned." Learning your native language is not at all the same task as translating between languages. People who do translation are usually quite knowledgable in questions of grammar. This is particularly true of people who pick up languages later in life. Even if you know multiple languages "natively", it's often difficult to translate accurately without thinking about grammar. Or, for that matter, to speak/write your own language with any degree of competency -- we study grammar in grade school for a reason.
- 10y ago

Eridrus 10y ago

We tried solving this by analysing language for decades; the reason we're doing this now is because it actually works a lot better.

timr 10y ago
The article directly suggests that isn't the case. There's a (debatable) improvement over Google's existing phrase-based system on these benchmarks, but the whole paper is about the huge effort they've had to put in to get past the shortcomings of the method. A more realistic assessment is that state-of-the-art neural translation is maybe getting as good as phrase-based systems.
- Houshalter 10y ago
  They claim 60% fewer translation errors than the phrase based system. That's not nothing.
  - Houshalter 10y ago
    Also, see this graph: https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAABPg/-6OhKKPhxT89Vs9HhyKMEnyG_0ncWGjJQCLcB/s1600/image00.png https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAAB... On French > English and English > Spanish it's almost indistinguishable from human translations. On the rest it still gets greater than half the distance between the previous best method and human quality translations. This is a massive step forward.
    - timr 10y ago
      "This is a massive step forward." Take it easy. The graph you're linking to is for a limited survey of 500 isolated sentences, and it's a comparison of mean scores without confidence intervals. The authors themselves admit that it has serious problems as an evaluative tool: "Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work." When you look at the other evaluations in the paper, it's clear that it's about as good as the current best phrase-based tools. In other words, this may be a massive step forward for neural nets, but it puts them at about par for the field.

emcq 10y ago

Christopher Manning, a somewhat famous NLP researcher before and during the deep learning craze, has a great perspective on this that echoes your statements which I recommend reading: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239 http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239

SamPhillips 10y ago

This reminds me of the DeepMind bit where the AI beat human players despite not knowing about the edifice of built up knowledge in that domain. I'd expect that an AI could be a great chess player with enough gameplay info and not knowing particular strength scoring, types of famous defenses, etc. In some ways the way that humans have constructed these relationships may actually make learning more difficult. We certainly don't teach 3-year olds about theta criterion, we just speak in 'correct' language and may correct them when they make mistakes.

dmreedy 10y ago

My hunch is that the more apt analogy is to Skinnerian Behaviorists, or apostles of the Hilbert Program. Deep learning is a very a powerful tool, and the current zeal for it has yielded rapid progress in previously stagnating fields. That said, we have yet to see what its limits are in this space; it's still too young to know. It may be that, like in so many cases beforehand, synthesis is the path forward, and there are no silver bullets.

blackkettle 10y ago

Why do think any of those labels are critically important? They are entirely derived from human desire to understand, process, and define a taxonomy for language. There is absolutely no reason to think that they are inherently necessary to achieve the goal or high fidelity translation (or anything else in this domain). Similarly they might be meaningful but inferred naturally through the learning process. If your goal is to map machine learning processes to established dogma, this makes sense; if it is just to build an effective learning machine that solves an extant problem, then it's not.

d_burfoot 10y ago

Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observation: 1) Linguistic knowledge and theory is not useful for NLP work; instead, people should rely entirely on machine learning techniques. 2) Linguistic knowledge is useful in principle, but the field of linguistics has not yet obtained a sufficiently high-quality theory, and it is better to rely on ML than on low-quality theory. Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

sushirain 10y ago
The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition. Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.
sidarape 10y ago
> it is better to rely on ML than on low-quality theory. Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.
- Isamu 10y ago
  In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity. But I agree, it is a not a very explanitory theory since it is too black-box.
gradys 10y ago
I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper. It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this. That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML. There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model. [1] - http://arxiv.org/abs/1603.06021 http://arxiv.org/abs/1603.06021

raverbashing 10y ago

I guess there are two issues here: - Most of the available texts have some form of "errors" or are not grammatically complete - Machine translation does not care about grammar, it cares about probabilities NLP with grammar knowledge is feature engineering basically. It is useful when you're "shallow learning" but not so much when you're deep learning

eva1984 10y ago

Linguistics fails to come up with a competitive alternative that leads us to better real application, that is why. In a data driven world, that is nothing to complain about except themselves.

mtrn 10y ago

> There is nothing about: part of speech categories, relative clauses, morphology This is an interesting and deep question. I believe that there are multiple ways to tell the story of a phenomenon. Using POS categories or verb tenses helps us to organize and learn about the medium. It is important for humans to know what a verb tense - in theory and practice - is, to be able to form not just grammatical sentences, but also ones, that express things clearer. However, there is no rule that says that these concepts must be learned first before you can do anything else. In a sense, I see the (neural) machines taking a step back. Whereas earlier attempts at language modeling took into account a lot of linguistic concepts, maybe even tailored to a specific human language, the more end-to-end approaches ignore these things and get better results. What can this tell you about the concepts? They are certainly useful and have been established by a scientific community. The machine, as it reads not just tens or hundreds of sentences but billions, creates a mathematical model, that happens to have aspects, that have very useful real world sides to them. A machine that learns a language model - or that learns translation, can teach us new things about words, and ultimately also about the world. I believe this is a beautiful aspect of the current neural renaissance. ---- Kanerva in a preface to Geometry and Meaning writes: > We think of some sciences, by their very nature, as being more mathematical than others. We even refer to physics and other heavily mathematical sciences as "hard sciences," as if to imply that social sciences and humanities are somehow soft or easy. Another way to see this picture, however, is that mathematics itself is hard and that it would be applied first to sciences that themselves are relatively easy. [...] It matters greatly that the mathematics be appropriate. [...] Mathematics is a tool for coping with complexity. [...] This brings us to the substance of [this] book: the exploration of mathematics that would be appropriate for describing concepts and meaning. The branch of mathematics that has traditionally been associated with meaning is logic. Logic is also discussed here, yet the main emphasis is on identifying mathematical spaces that would lend to modeling of meaning. (http://www.puttypeg.net/book/chapters/foreword.html http://www.puttypeg.net/book/chapters/foreword.html)

arikrak 10y ago

Actually it seems a little like the opposite. Their software is actually translating language quite well in the real empirical world, while the the programs built with language rules were never able to do nearly as well.

joseraul 10y ago

Peter Norvig explains why. http://norvig.com/chomsky.html http://norvig.com/chomsky.html

visarga 10y ago

> There is nothing about: part of speech categories, relative clauses, ... Because it's not the best level of abstraction. Human generated features don't capture all the nuances. In vision there used to be heavy engineering and math involved in feature creation. Now it's all neural nets end to end. Same thing happened to language.

nl 10y ago

Google's recent language modelling paper achieves state of the art performance on exactly these tasks, and it's based on a somewhat vaguely similar architecture. It's becoming pretty clear that many of these concepts may not be fundamental 'things' but just things that make it easy to talk about.

AstralStorm 10y ago

No seriously, this is exactly empirical work. It works somewhat at translating texts. On the other hand, attempts that tried to use linguistic categories failed. A nice new development is multilevel deep networks with explanatory states built in. It will tell you why it detected the thing it did. It may of course be lying or wrong. :)