6 ms·

SyntaxNet in Context: Understanding Google's New TensorFlow NLP Model

183 points by syllogism 10y ago

nxzero 10y ago
>> "On the time-honoured benchmark for this task" Anyone know what exactly this benchmark was, it's relevance, and if it's been independently reproduced?
- LanguageGamer 10y ago
  According to the paper linked by the original announcement [1], the parser scores 94.41% for unlabeled attachment on the Wall Street Journal corpus [2], a parsed and labeled data set of 30 million words. This corpus is a standard for NLP research on english syntax, but I think its worth remembering there is a great deal of disagreement among linguists about what the syntax of english is and what the lexical categories are. [1] http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html?m=1 http://googleresearch.blogspot.com/2016/05/announcing-syntax... [2] https://catalog.ldc.upenn.edu/LDC2000T43 https://catalog.ldc.upenn.edu/LDC2000T43
deleted 10y ago
[deleted]
projectramo 10y ago
Wouldn't syntactic parsing also improve by more knowledge of the real world? From the example in the text: "They ate the pizza with anchovies" Isn't it important to know that humans don't use anchovies to eat pizza? You need some semantics as well. Otherwise, from pure syntax, how can you tell it apart from: "They ate the pizza with forks"
- syllogism 10y ago
  I guess I should've connected the dots a bit better on this. The neural network model offers a pretty compelling answer to this. The idea is that you each word is represented as a real-valued vector, and these vectors encode information about the word's meaning. So, "pizza" encodes in it that it's a type of food, and so does "anchovies". But "fork" is closer to "knife" and "spoon". Linear models struggle with this. In a linear model, mostly, each word is an island. "fork" and "spoon" each have unique IDs, and so you'd better hope you see enough examples of each of them to understand what they mean. It helps to add semantic cluster IDs to the words as well, but this isn't quite as good as the word vectors. What doesn't work particularly well is encoding external knowledge explicitly. Having a semantic resource that tells you explicitly that "fork" and "spoon" are members of class "cutlery". This has been tried by many people, and it produces small or no benefit.
  - danieldk 10y ago
    Linear models struggle with this. In a linear model, mostly, each word is an island. It's actually quite easy to do this with a linear model. You use the distribution of head/relation/dependent pairs as an auxiliary distribution in a linear model. This has been done for quite some time (see my answer to your parent). This auxiliary distribution is then estimated using machine-annotated data. Of course, a non-linear classifier using embeddings is cheaper to train ;), because you don't have to parse a lot of data ahead of time.
    - visarga 10y ago
      This kind of representation is famously done by word2vec, also from Google, and the Python package Gensim. Many NLP projects start directly from word embeddings precomputed on very large corpora of text.
      - danieldk 10y ago
        No, this is not the type of representation that is done by word2vec. These older bilexical preference models work as follows: - Take a huge corpus. - Parse the corpus using your parser. - Extract head-dependent strengths according to some metric (e.g. pointwise mutual information) from the machine annotated data. These association strengths are used as an auxiliary distribution. The machine annotated data may contain erroneous analyses, but they are typically outnumbered by correct analyses[1]. - In parse disambiguation, the head-dependent strength is added as another feature (the strength is the feature value). This idea precedes word2vec for quite some time and is also different, both in the training procedure (word2vec uses raw unannotated data, bilexical preference models use machine-annotation data) as well as the result of training (distributed word representations vs. head-dependent association strengths). Although it's a different method, they can capture the same thing as word embeddings plus a non-linear classifier: which heads typically take which dependents. However, in contrast to word embeddings, they are effective in a linear classifier because the auxiliary distribution itself consists of combinatory features (head-dependents). [1] E.g. consider freer word order languages. In Dutch, SVO is the preferred word order, but OVS is also permitted. A parser without such an auxiliary distribution might analyse the object as a subject in an OVS sentence. However, since SVO is much more frequent than OVS, it will learn the correct associations on a large machine-annotated corpus.
- shaynesweeney 10y ago
  To your point, how does NLP address grammatical differences/errors, colloquialisms or slang? In the context of personal assistants every implementation I've seen has required more unnatural-language given the status quo.
- benjaminl 10y ago
  This is addressed in the original announcement post of SyntaxNet[0]: The major source of errors at this point are examples such as the prepositional phrase attachment ambiguity described above, which require real world knowledge (e.g. that a street is not likely to be located in a car) and deep contextual reasoning. Machine learning (and in particular, neural networks) have made significant progress in resolving these ambiguities. But our work is still cut out for us: we would like to develop methods that can learn world knowledge and enable equal understanding of natural language across all languages and contexts. The SyntaxNet team thinks that this is indeed a major source of errors and seems to be a focus of their work going forward. [0] - http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html http://googleresearch.blogspot.com/2016/05/announcing-syntax...
  - projectramo 10y ago
    Honestly surprised that they use Neural Networks for this because I don't know how they can reason about facts. I know, this is an ancient debate which I don't want to reanimate. Maybe the field has moved on from when I was up to date on matters.
    - deleted 10y ago
      [deleted]
- danieldk 10y ago
  You can learn this quite easily using bilexical dependencies on a machine annotated corpus, even without word embeddings (which is another option as Matthew suggests in a sibling post): http://www.aclweb.org/anthology/W07-2201 http://www.aclweb.org/anthology/W07-2201
- thrownaway2424 10y ago
  Wasn't this the kind of understanding that Cyc was supposed to accomplish?
- amelius 10y ago
  Another way to parse this would be similar to: "They ate the pizza with friends"
  - bronson 10y ago
    They ate the pizza with excitement.
  - igravious 10y ago
    "They ate the pizza with friends looking on in disgust"
fizzbatter 10y ago
Somewhat off topic, but for anyone more familiar with spaCy: Any idea if spaCy has a sane way to be used in other languages? Specifically Rust, in this case? I'm in need of a decent NLP library to train and or map sentences to intents. Somewhat similar to what wit.ai offers, but it needs to be offline-able. spaCy sounds great, but requiring a runtime (if spaCy indeed does) on the installed system is a no-go for me. Not knowing much about Python, It looks like it is actually compatible with CPython, so could it therefor be compiled to C and imported into Rust?
- syllogism 10y ago
  spaCy's implemented in Cython, which compiles into C++. So, the end-state at the moment is a .so object that expects to be loaded into Python. Now, you can instead tell Cython to compile into standalone objects, so that they can be imported from C/C++ code. I'm not well versed on this, but if you can call into C/C++ libraries easily from Rust, you should be able to call into spaCy. You'll miss some of the niceties from the Python layer, but the C-level API is pretty well behaved. So, you should be able to get by okay.
  - danieldk 10y ago
    Curious: since spaCy is a commercial project (I know that it is open source), did you consider writing it in C or C++, it would make binding to other languages a lot easier. I can imagine that importing the Python runtime can be problematic in some projects. Also, I assume that you will see some Python types in the API when you want to call the parser from C? (My own NN dependency parser is written in Go, since it's mostly for research, but if I wanted to make it embeddable, I'd have implemented it in C++ or Rust with a C API.)
    - syllogism 10y ago
      Well, I like writing in Cython, and I figured I was better off pleasing some people (Python) a lot before trying to please a lot of people a little. It won't be hard to port to C++ when the time comes.
- 10y ago
visarga 10y ago
Haha. It's like AST (abstract syntax tree), but for human language. We just need the code generator to make human language run on CPUs. </joking>

183 points by syllogism 10y ago

>> "On the time-honoured benchmark for this task" Anyone know what exactly this benchmark was, it's relevance, and if it's been independently reproduced?

LanguageGamer 10y ago
According to the paper linked by the original announcement [1], the parser scores 94.41% for unlabeled attachment on the Wall Street Journal corpus [2], a parsed and labeled data set of 30 million words. This corpus is a standard for NLP research on english syntax, but I think its worth remembering there is a great deal of disagreement among linguists about what the syntax of english is and what the lexical categories are. [1] http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html?m=1 http://googleresearch.blogspot.com/2016/05/announcing-syntax... [2] https://catalog.ldc.upenn.edu/LDC2000T43 https://catalog.ldc.upenn.edu/LDC2000T43

deleted 10y ago

[deleted]

projectramo 10y ago

Wouldn't syntactic parsing also improve by more knowledge of the real world? From the example in the text: "They ate the pizza with anchovies" Isn't it important to know that humans don't use anchovies to eat pizza? You need some semantics as well. Otherwise, from pure syntax, how can you tell it apart from: "They ate the pizza with forks"

syllogism 10y ago
I guess I should've connected the dots a bit better on this. The neural network model offers a pretty compelling answer to this. The idea is that you each word is represented as a real-valued vector, and these vectors encode information about the word's meaning. So, "pizza" encodes in it that it's a type of food, and so does "anchovies". But "fork" is closer to "knife" and "spoon". Linear models struggle with this. In a linear model, mostly, each word is an island. "fork" and "spoon" each have unique IDs, and so you'd better hope you see enough examples of each of them to understand what they mean. It helps to add semantic cluster IDs to the words as well, but this isn't quite as good as the word vectors. What doesn't work particularly well is encoding external knowledge explicitly. Having a semantic resource that tells you explicitly that "fork" and "spoon" are members of class "cutlery". This has been tried by many people, and it produces small or no benefit.
- danieldk 10y ago
  Linear models struggle with this. In a linear model, mostly, each word is an island. It's actually quite easy to do this with a linear model. You use the distribution of head/relation/dependent pairs as an auxiliary distribution in a linear model. This has been done for quite some time (see my answer to your parent). This auxiliary distribution is then estimated using machine-annotated data. Of course, a non-linear classifier using embeddings is cheaper to train ;), because you don't have to parse a lot of data ahead of time.
  - visarga 10y ago
    This kind of representation is famously done by word2vec, also from Google, and the Python package Gensim. Many NLP projects start directly from word embeddings precomputed on very large corpora of text.
    - danieldk 10y ago
      No, this is not the type of representation that is done by word2vec. These older bilexical preference models work as follows: - Take a huge corpus. - Parse the corpus using your parser. - Extract head-dependent strengths according to some metric (e.g. pointwise mutual information) from the machine annotated data. These association strengths are used as an auxiliary distribution. The machine annotated data may contain erroneous analyses, but they are typically outnumbered by correct analyses[1]. - In parse disambiguation, the head-dependent strength is added as another feature (the strength is the feature value). This idea precedes word2vec for quite some time and is also different, both in the training procedure (word2vec uses raw unannotated data, bilexical preference models use machine-annotation data) as well as the result of training (distributed word representations vs. head-dependent association strengths). Although it's a different method, they can capture the same thing as word embeddings plus a non-linear classifier: which heads typically take which dependents. However, in contrast to word embeddings, they are effective in a linear classifier because the auxiliary distribution itself consists of combinatory features (head-dependents). [1] E.g. consider freer word order languages. In Dutch, SVO is the preferred word order, but OVS is also permitted. A parser without such an auxiliary distribution might analyse the object as a subject in an OVS sentence. However, since SVO is much more frequent than OVS, it will learn the correct associations on a large machine-annotated corpus.
shaynesweeney 10y ago
To your point, how does NLP address grammatical differences/errors, colloquialisms or slang? In the context of personal assistants every implementation I've seen has required more unnatural-language given the status quo.
benjaminl 10y ago
This is addressed in the original announcement post of SyntaxNet[0]: The major source of errors at this point are examples such as the prepositional phrase attachment ambiguity described above, which require real world knowledge (e.g. that a street is not likely to be located in a car) and deep contextual reasoning. Machine learning (and in particular, neural networks) have made significant progress in resolving these ambiguities. But our work is still cut out for us: we would like to develop methods that can learn world knowledge and enable equal understanding of natural language across all languages and contexts. The SyntaxNet team thinks that this is indeed a major source of errors and seems to be a focus of their work going forward. [0] - http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html http://googleresearch.blogspot.com/2016/05/announcing-syntax...
- projectramo 10y ago
  Honestly surprised that they use Neural Networks for this because I don't know how they can reason about facts. I know, this is an ancient debate which I don't want to reanimate. Maybe the field has moved on from when I was up to date on matters.
  - deleted 10y ago
    [deleted]
danieldk 10y ago
You can learn this quite easily using bilexical dependencies on a machine annotated corpus, even without word embeddings (which is another option as Matthew suggests in a sibling post): http://www.aclweb.org/anthology/W07-2201 http://www.aclweb.org/anthology/W07-2201
thrownaway2424 10y ago
Wasn't this the kind of understanding that Cyc was supposed to accomplish?
amelius 10y ago
Another way to parse this would be similar to: "They ate the pizza with friends"
- bronson 10y ago
  They ate the pizza with excitement.
- igravious 10y ago
  "They ate the pizza with friends looking on in disgust"

fizzbatter 10y ago

Somewhat off topic, but for anyone more familiar with spaCy: Any idea if spaCy has a sane way to be used in other languages? Specifically Rust, in this case? I'm in need of a decent NLP library to train and or map sentences to intents. Somewhat similar to what wit.ai offers, but it needs to be offline-able. spaCy sounds great, but requiring a runtime (if spaCy indeed does) on the installed system is a no-go for me. Not knowing much about Python, It looks like it is actually compatible with CPython, so could it therefor be compiled to C and imported into Rust?

syllogism 10y ago
spaCy's implemented in Cython, which compiles into C++. So, the end-state at the moment is a .so object that expects to be loaded into Python. Now, you can instead tell Cython to compile into standalone objects, so that they can be imported from C/C++ code. I'm not well versed on this, but if you can call into C/C++ libraries easily from Rust, you should be able to call into spaCy. You'll miss some of the niceties from the Python layer, but the C-level API is pretty well behaved. So, you should be able to get by okay.
- danieldk 10y ago
  Curious: since spaCy is a commercial project (I know that it is open source), did you consider writing it in C or C++, it would make binding to other languages a lot easier. I can imagine that importing the Python runtime can be problematic in some projects. Also, I assume that you will see some Python types in the API when you want to call the parser from C? (My own NN dependency parser is written in Go, since it's mostly for research, but if I wanted to make it embeddable, I'd have implemented it in C++ or Rust with a C API.)
  - syllogism 10y ago
    Well, I like writing in Cython, and I figured I was better off pleasing some people (Python) a lot before trying to please a lot of people a little. It won't be hard to port to C++ when the time comes.
10y ago

visarga 10y ago

Haha. It's like AST (abstract syntax tree), but for human language. We just need the code generator to make human language run on CPUs. </joking>