4 ms·

> Why is NLP Hard? ... Language is highly ambiguous - it relies on subtle cues and contexts to convey meaning. This is true, but it is only part of the answer.

> Why is NLP Hard? ... Language is highly ambiguous - it relies on subtle cues and contexts to convey meaning. This is true, but it is only part of the answer. Another part of the answer is what I call the Long Tail of Grammar. It turns out that if you try to write down all the rules of grammar, you will not get 40 or 60 rules, but something more like 100s or maybe even 1000s of rules. Most of those rules are obscure, rare, archaic, or useable only in specific contexts or with specific words. However, they are part of the language, a native speaker will be able to use them and comprehend them without difficulty, and an NLP system must be able to "understand" them in order to extract the correct meaning from a sentence. As just a minor example off the top of my head, compare the phrase "peeled peach" with "hairy-peeled peach". The former phrase means a peach without a peel, while the latter means a peach with a hairy peel. So a good NLP system must not only recognize the existence of the two grammatical rules, but also be able to disambiguate them correctly.

will_pseudonym 10y ago
Not just that, grammar rules can come into existence/change spontaneously. Here's an example and name for this phenomenon: Because syntactic change.
- WildUtah 10y ago
  That isn't really a rule change. The lexicon just added a new preposition, 'because,' to the old list. It means almost the same thing as the old preposition 'because of.' There's a difference between coining a new word and changing a grammar rule.
  - will_pseudonym 10y ago
    No, it is a rule change. It's turning a sentence fragment into a sentence.
chch 10y ago
> or useable only in specific contexts or with specific words. A good example of this is the Winograd Schema. You might think you can figure out a good algorithm for anaphoric resolution (i.e. If you see "Sally called and she said hello.", who is "she"?) that just relies on the structure of a sentence, without considering semantics. But here's a counterexample: "The city councilmen refused the demonstrators a permit because they feared violence." Who are 'they'? "The city councilmen refused the demonstrators a permit because they advocated violence." Now who are 'they'? If you're like most people, even though only the verb changed, the binding of 'they' based on the deeper semantic meaning also changed. These sentences are called Winograd Schema[1], and there are plenty more like it. [1] https://en.wikipedia.org/wiki/Winograd_Schema_Challenge https://en.wikipedia.org/wiki/Winograd_Schema_Challenge
- ice109 10y ago
  re the council people sentences: I don't understand the problem. they're ill-defined sentences. we use heuristics to parse them but those heuristics can fail (the council denied the demonstrators permit because they feared violence... and the council was obliging). just teach the computer the heuristics like we learn them.
  - chch 10y ago
    That's exactly the issue. The way we learn them is through world experience, which is sometimes hard to figure out how to transfer into a computer. Example: "I dropped the egg on my glass living room table and it broke!" "I dropped my hammer on my glass living room table and it broke!" These are both ill-defined semantically, but if you asked most native English speakers "what broke" for each sentence, they'd probably say "egg" for the first and "table" for the second. It could be the other, but it would be surprising. So, to solve just the "Dropped X on Y, Z broke" problem, we'd need to teach the computer to understand the effect of the relative 'fragility scores' of each object. Personally, I never sat down and memorized a chart of these as a human. You could perhaps use machine learning to derive the data by analyzing a large corpus of text[1], and match humans most of the time, but then that's just one sentence type solved, out of any number of other tricky constructions. So the long tail of semantic understanding quickly becomes a very fun set of problems to solve, for certain definitions of fun. :) A few more examples to consider how you would teach a computer to understand, from a Winograd Schema corpus[2]: John couldn't see the stage with Billy in front of him because he is so [short/tall]. Who is so [short/tall]? The sculpture rolled off the shelf because it wasn't [anchored/level]. What wasn't [anchored/level]? The older students were bullying the younger ones, so we [rescued/punished] them. Whom did we [rescue/punish]? I tried to paint a picture of an orchard, with lemons in the lemon trees, but they came out looking more like [light bulbs / telephone poles]. What looked like [light bulbs / telephone poles]? [1] e.g. http://cs.rochester.edu/research/lore/ http://cs.rochester.edu/research/lore/ [2] http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.html http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/...
leereeves 10y ago
> compare the phrase "peeled peach" with "hairy-peeled peach" Is that a rule of grammar, or simply the meanings of the adjectives "peeled" and "hairy-peeled"?
- andrewtbham 10y ago
  yeh it seems more like another example of word sense disambiguation.
- d_burfoot 10y ago
  In my mind there are two very distinct patterns: - Xed Y = a Y that has been Xed, where X is a verb - Xed Y = a Y that is equipped with an X, where X is a noun. For the latter pattern, consider examples such as "red-lipped woman", "rosy-fingered dawn", "sharp-toothed pike", "horned owl", etc.
argonaut 10y ago
Your example isn't even a specific grammar rule. It's a very broad language rule, which is: "descriptions should conform to the real world." Someone who has never seen a peach might assume that hairy peeled peach refers to a hairy peach without a peel. If you've seen a peach before, this makes no sense. So the assumption is that the peach has a hairy peel. This is why purely statistical / supervised learning-based NLP alone is not enough.
throwaway91111 10y ago
This seems like a great argument for automatic grammar learning. How far has current research taken us in that direction? I'd much rather feed it edge cases to accommodate than rules to follow.
- hiddencost 10y ago
  They tried that in the 70s and 80s until they realized they were wasting their time. Probabilistic context free grammars were a thing, also. Long story short, there's no such thing as grammar but they're a nice fiction for talking about communication. The deeper you get into NLP the more you (1) see what jelinek was talking about when he said "every time I fire a linguist, accuracy goes up" (he was hiring physicists and information theorists), and (2) realize that basically every thought, belief, and statement is deeply ambiguous, and that most human communication is ad hoc. Also, the more time you spend looking at live data from users, the more you realize that the notion of language as a generally shared system of meaning is not real. Trivial communication about basic tasks is doable, although you often will fuck up there too when talking to someone with deeply different cultural expectations. If you think that _any_ sentence has unambiguous meaning, you should try to meet some people who are more different than you are. Or get into deeper conversations with the people you know. Edit: typo
  - ktRolster 10y ago
    there's no such thing as grammar I'm not sure what you mean by this. Certainly adjectives are a thing, and if I learn a new adjective, "anguilliform" for example, I have never heard that in context, but I know exactly how to use it. That is grammar, right?
    - hiddencost 10y ago
      I'm being hyperbolic, but what I'm suggesting is that grammar is a convenient fiction. The part where you speak of "how to use it" points to the break down in your thinking. (Side note: look into the difference between prescriptive and descriptive linguistics for a sense of where I'm coming from on that point.) So, you learn a new adjective. Surely you can use it like any other adjective, right? Sure. But someone can also very comfortably use it in a way that violates your notions of how adjectives work, and you'll probably know what they mean. Or you'll use it in a context that makes sense to you, but not to someone you're talking to. The way to use language is in the way that allows others to understand you. Paradoxically that doesn't actually mean adhering to some arbitrary set of rules. Here's an important paper [0]. Basic idea: you get a small community isolated, and they just rip the rules to shreds, but what they end up building is often much higher bandwidth and allows for more complex ideas. The point I'm pushing on is this: engineers especially think of language as an agreed upon set of rules which can be used correctly or incorrectly. Turns out that in practice it's a chaotic mess of individuals who abuse the rules mercilessly with minimal regard for how they're supposed to use it, and still get along happily. Developing an understanding of the inherent fuzziness of words and structure in communication can actually help a person develop significant capacity for self expression. The failure to understand this is one of the reasons most engineers write shitty poetry. :P [0] http://onlinelibrary.wiley.com/doi/10.1111/1467-9481.00177/full http://onlinelibrary.wiley.com/doi/10.1111/1467-9481.00177/f...

will_pseudonym 10y ago

Not just that, grammar rules can come into existence/change spontaneously. Here's an example and name for this phenomenon: Because syntactic change.

WildUtah 10y ago
That isn't really a rule change. The lexicon just added a new preposition, 'because,' to the old list. It means almost the same thing as the old preposition 'because of.' There's a difference between coining a new word and changing a grammar rule.
- will_pseudonym 10y ago
  No, it is a rule change. It's turning a sentence fragment into a sentence.

chch 10y ago

> or useable only in specific contexts or with specific words. A good example of this is the Winograd Schema. You might think you can figure out a good algorithm for anaphoric resolution (i.e. If you see "Sally called and she said hello.", who is "she"?) that just relies on the structure of a sentence, without considering semantics. But here's a counterexample: "The city councilmen refused the demonstrators a permit because they feared violence." Who are 'they'? "The city councilmen refused the demonstrators a permit because they advocated violence." Now who are 'they'? If you're like most people, even though only the verb changed, the binding of 'they' based on the deeper semantic meaning also changed. These sentences are called Winograd Schema[1], and there are plenty more like it. [1] https://en.wikipedia.org/wiki/Winograd_Schema_Challenge https://en.wikipedia.org/wiki/Winograd_Schema_Challenge

ice109 10y ago
re the council people sentences: I don't understand the problem. they're ill-defined sentences. we use heuristics to parse them but those heuristics can fail (the council denied the demonstrators permit because they feared violence... and the council was obliging). just teach the computer the heuristics like we learn them.
- chch 10y ago
  That's exactly the issue. The way we learn them is through world experience, which is sometimes hard to figure out how to transfer into a computer. Example: "I dropped the egg on my glass living room table and it broke!" "I dropped my hammer on my glass living room table and it broke!" These are both ill-defined semantically, but if you asked most native English speakers "what broke" for each sentence, they'd probably say "egg" for the first and "table" for the second. It could be the other, but it would be surprising. So, to solve just the "Dropped X on Y, Z broke" problem, we'd need to teach the computer to understand the effect of the relative 'fragility scores' of each object. Personally, I never sat down and memorized a chart of these as a human. You could perhaps use machine learning to derive the data by analyzing a large corpus of text[1], and match humans most of the time, but then that's just one sentence type solved, out of any number of other tricky constructions. So the long tail of semantic understanding quickly becomes a very fun set of problems to solve, for certain definitions of fun. :) A few more examples to consider how you would teach a computer to understand, from a Winograd Schema corpus[2]: John couldn't see the stage with Billy in front of him because he is so [short/tall]. Who is so [short/tall]? The sculpture rolled off the shelf because it wasn't [anchored/level]. What wasn't [anchored/level]? The older students were bullying the younger ones, so we [rescued/punished] them. Whom did we [rescue/punish]? I tried to paint a picture of an orchard, with lemons in the lemon trees, but they came out looking more like [light bulbs / telephone poles]. What looked like [light bulbs / telephone poles]? [1] e.g. http://cs.rochester.edu/research/lore/ http://cs.rochester.edu/research/lore/ [2] http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.html http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/...

leereeves 10y ago

> compare the phrase "peeled peach" with "hairy-peeled peach" Is that a rule of grammar, or simply the meanings of the adjectives "peeled" and "hairy-peeled"?

andrewtbham 10y ago
yeh it seems more like another example of word sense disambiguation.
d_burfoot 10y ago
In my mind there are two very distinct patterns: - Xed Y = a Y that has been Xed, where X is a verb - Xed Y = a Y that is equipped with an X, where X is a noun. For the latter pattern, consider examples such as "red-lipped woman", "rosy-fingered dawn", "sharp-toothed pike", "horned owl", etc.

argonaut 10y ago

Your example isn't even a specific grammar rule. It's a very broad language rule, which is: "descriptions should conform to the real world." Someone who has never seen a peach might assume that hairy peeled peach refers to a hairy peach without a peel. If you've seen a peach before, this makes no sense. So the assumption is that the peach has a hairy peel. This is why purely statistical / supervised learning-based NLP alone is not enough.

throwaway91111 10y ago

This seems like a great argument for automatic grammar learning. How far has current research taken us in that direction? I'd much rather feed it edge cases to accommodate than rules to follow.

hiddencost 10y ago
They tried that in the 70s and 80s until they realized they were wasting their time. Probabilistic context free grammars were a thing, also. Long story short, there's no such thing as grammar but they're a nice fiction for talking about communication. The deeper you get into NLP the more you (1) see what jelinek was talking about when he said "every time I fire a linguist, accuracy goes up" (he was hiring physicists and information theorists), and (2) realize that basically every thought, belief, and statement is deeply ambiguous, and that most human communication is ad hoc. Also, the more time you spend looking at live data from users, the more you realize that the notion of language as a generally shared system of meaning is not real. Trivial communication about basic tasks is doable, although you often will fuck up there too when talking to someone with deeply different cultural expectations. If you think that _any_ sentence has unambiguous meaning, you should try to meet some people who are more different than you are. Or get into deeper conversations with the people you know. Edit: typo
- ktRolster 10y ago
  there's no such thing as grammar I'm not sure what you mean by this. Certainly adjectives are a thing, and if I learn a new adjective, "anguilliform" for example, I have never heard that in context, but I know exactly how to use it. That is grammar, right?
  - hiddencost 10y ago
    I'm being hyperbolic, but what I'm suggesting is that grammar is a convenient fiction. The part where you speak of "how to use it" points to the break down in your thinking. (Side note: look into the difference between prescriptive and descriptive linguistics for a sense of where I'm coming from on that point.) So, you learn a new adjective. Surely you can use it like any other adjective, right? Sure. But someone can also very comfortably use it in a way that violates your notions of how adjectives work, and you'll probably know what they mean. Or you'll use it in a context that makes sense to you, but not to someone you're talking to. The way to use language is in the way that allows others to understand you. Paradoxically that doesn't actually mean adhering to some arbitrary set of rules. Here's an important paper [0]. Basic idea: you get a small community isolated, and they just rip the rules to shreds, but what they end up building is often much higher bandwidth and allows for more complex ideas. The point I'm pushing on is this: engineers especially think of language as an agreed upon set of rules which can be used correctly or incorrectly. Turns out that in practice it's a chaotic mess of individuals who abuse the rules mercilessly with minimal regard for how they're supposed to use it, and still get along happily. Developing an understanding of the inherent fuzziness of words and structure in communication can actually help a person develop significant capacity for self expression. The failure to understand this is one of the reasons most engineers write shitty poetry. :P [0] http://onlinelibrary.wiley.com/doi/10.1111/1467-9481.00177/full http://onlinelibrary.wiley.com/doi/10.1111/1467-9481.00177/f...