5 ms·

Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observatio

Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observation: 1) Linguistic knowledge and theory is not useful for NLP work; instead, people should rely entirely on machine learning techniques. 2) Linguistic knowledge is useful in principle, but the field of linguistics has not yet obtained a sufficiently high-quality theory, and it is better to rely on ML than on low-quality theory. Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

sushirain 10y ago
The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition. Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.
sidarape 10y ago
> it is better to rely on ML than on low-quality theory. Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.
- Isamu 10y ago
  In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity. But I agree, it is a not a very explanitory theory since it is too black-box.
gradys 10y ago
I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper. It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this. That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML. There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model. [1] - http://arxiv.org/abs/1603.06021 http://arxiv.org/abs/1603.06021
- MarkMc2412 10y ago
  Do you have any further detail or reference to Wittgenstein here? Curious to learn more
  - thegreatestape 10y ago
    Not OP, but Wittgenstein covers this in his Philosophical Investigations. In the Tractatus, he tried to create an all-encompassing system for language, but in PI he more or less says it's impossible. It's an amazing read in any case, it has totally changed the way I approach philosophy.
    - MarkMc2412 10y ago
      Ok, thanks!
- j-pb 10y ago
  > which benefit from the linguistic insight that language has recursive structure. Really? You needed an entire _field_, and not just, and by just I mean, in an, but not too, extended, sense, really, trivial observation, to get to that conclusion?
murbard2 10y ago
I love your work on #2, it's a really interesting avenue, but I wonder if both approaches are close to the point where reasoning about the sentence and its meaning becomes necessary anyway if we want to go beyond the current state of the art.
kafkaesq 10y ago
Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true. "Superior ML performance as a pain signal to be carefully listened to", as it were. I'm in camp #2, also.

sushirain 10y ago

The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition. Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.

sidarape 10y ago

> it is better to rely on ML than on low-quality theory. Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.

Isamu 10y ago
In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity. But I agree, it is a not a very explanitory theory since it is too black-box.

gradys 10y ago

I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper. It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this. That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML. There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model. [1] - http://arxiv.org/abs/1603.06021 http://arxiv.org/abs/1603.06021

MarkMc2412 10y ago
Do you have any further detail or reference to Wittgenstein here? Curious to learn more
- thegreatestape 10y ago
  Not OP, but Wittgenstein covers this in his Philosophical Investigations. In the Tractatus, he tried to create an all-encompassing system for language, but in PI he more or less says it's impossible. It's an amazing read in any case, it has totally changed the way I approach philosophy.
  - MarkMc2412 10y ago
    Ok, thanks!
j-pb 10y ago
> which benefit from the linguistic insight that language has recursive structure. Really? You needed an entire _field_, and not just, and by just I mean, in an, but not too, extended, sense, really, trivial observation, to get to that conclusion?

murbard2 10y ago

I love your work on #2, it's a really interesting avenue, but I wonder if both approaches are close to the point where reasoning about the sentence and its meaning becomes necessary anyway if we want to go beyond the current state of the art.

kafkaesq 10y ago

Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true. "Superior ML performance as a pain signal to be carefully listened to", as it were. I'm in camp #2, also.