7 ms·
Evidence that this is the most accurate parser is here; the previous approach mentioned is a March 2016 paper, "Globally Normalized Transition-Based Neural Netw
by xigency 10y ago
Evidence that this is the most accurate parser is here;
the previous approach mentioned is a March 2016 paper, "Globally Normalized Transition-Based Neural Networks," http://arxiv.org/abs/1603.06042 http://arxiv.org/abs/1603.06042
"On a standard benchmark consisting of randomly drawn English newswire sentences (the 20 year old Penn Treebank), Parsey McParseface recovers individual dependencies between words with over 94% accuracy, beating our own previous state-of-the-art results, which were already better than any previous approach."
From the original paper, "Our model achieves state-of-the-art accuracy
on all of these tasks, matching or outperforming
LSTMs while being significantly faster.
In particular for dependency parsing on the Wall
Street Journal we achieve the best-ever published
unlabeled attachment score of 94.41%."
This seems like a narrower standard than described, specifically being better at parsing the Penn Treebank than the best natural language parser for English on the Wall Street Journal.
The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface.
- wodenokoto 10y agoThe paper you mention is the world's best results and is macparseface with broader beam search and more hidden layers. This is an opensourcing of the March 2016 method (syntaxnet, note that in the paper there are results from several trained models) as well as a trained model that is comparable in performance but faster (macparseface). It is very hard to separate those two things from the way they write.
- weinzierl 10y agospaCy is another active open source (MIT) POS-tagger. In a previous discussion on HN[1] it was well received. There is a simplified educational 200 lines python version [2] of it. It claims 96.8% for the WSJ corpus. What am I missing here? [1] https://news.ycombinator.com/item?id=8942783 https://news.ycombinator.com/item?id=8942783 [2] https://spacy.io/blog/part-of-speech-pos-tagger-in-python https://spacy.io/blog/part-of-speech-pos-tagger-in-python
- deleted 10y ago[deleted]
- kamkha 10y agospaCy's 96.8% accuracy is for the task of POS tagging while Google's reported 94% accuracy is for dependency parsing, a significantly harder problem.
- xigency 10y agoWell, it seems this page could use some work: http://www.aclweb.org/aclwiki/index.php?title=Dependency_Parsing_(State_of_the_art) http://www.aclweb.org/aclwiki/index.php?title=Dependency_Par...
- weinzierl 10y agoThanks (also to charlieegan3) this explains the difference nicely. Actually I found an educational 500 lines python parser from the same author as well [1]. It claims an accuracy of 92.7 for the WSJ corpus. [1] https://spacy.io/blog/parsing-english-in-python https://spacy.io/blog/parsing-english-in-python
- charlieegan3 10y agospaCy also has a dependency parser - looks like this blog post is just talking about the POS-tagger.
- syllogism 10y agoThose are the part-of-speech tag accuracies. spaCy's accuracy on the PTB evaluation is 92.2% --- so it makes 20% more errors than P. McP. On the other hand, spaCy is about 200x faster. I've been watching the line of research in SyntaxNet closely, and have been steadily working on replacing spaCy's averaged perceptron model with a neural network model. This is one of the main differences between spaCy and Parser McParseface. The key advantage of the neural network is that it lets you take advantage of training on lots and lots more text, in a semi-supervised way. In a linear model, you grow extra parameters when you do this. The neural network stays the same size --- it just gets better. So, you can benefit from reading the whole web into the neural network. This only works a little bit in the linear model, and it makes the resulting model enormous. Another difference is that spaCy is trained on whole documents, while P. McP. is trained in the standard set-up, using gold pre-processing. I speculate this will reduce the gap between the systems in a more realistic evlauation. Of course, P. McP can do the joint training too if they choose to. I've reached out to see whether they're interested in running the experiment: https://github.com/tensorflow/models/issues/65 https://github.com/tensorflow/models/issues/65
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- musesum 10y ago> The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface. So you're referring to this LSTM? "Andor et al. (2016)* is simply a SyntaxNet model with a larger beam and network. For futher information on the datasets, see that paper under the section "Treebank Union"." After spending a few months hand coding a NLP parser, am rather intrigued by LSTM. I like the idea of finding coefficients, as opposed to juggling artificial labels.
- xigency 10y agoYes, my mistake. Their claim is that SyntaxNet (originally described in the paper and improved over one month) is the best in field, whereas Parsey McParseface is just one trained instance.
- musesum 10y ago> SyntaxNet improved over one month ... whereas Parsey McParseface is just one trained instance Cool. I wonder how much how much Human effort (vs machine time) went into tweaking SyntaxNet versus tweaking Parsey M.?
- cbr 10y agobetter at parsing the Penn Treebank than the best natural language parser for English on the Wall Street Journal I'm pretty sure "the 20 year old Penn Treebank" and "the Wall Street Journal" are referring to the same dataset here. In the early 1990s the first large treebanking efforts were on a corpus from the WSJ, and they were released as the Penn Treebank: https://catalog.ldc.upenn.edu/LDC95T7 https://catalog.ldc.upenn.edu/LDC95T7 People report results on this dataset because that's what the field has been testing on (and overfitting to) for decades. (I worked on a successor project, OntoNotes, that involved additional treebank annotation on broader corpora: https://catalog.ldc.upenn.edu/LDC2013T19 https://catalog.ldc.upenn.edu/LDC2013T19)
- xigency 10y agoYes, the press release is (actually) pretty difficult to parse and really opaque in how the comparison is measured, which is why I wanted to throw into question the blog's headline, "The World's Most Accurate Parser." It seems more clear now but obviously Google doesn't feel the need to overtly prove that they are the best in the world at tasks, which is a bit questionable considering their number of followers. In all, it seems they have tested against several other dependency parsers, but clearly not all of them, and it's fair to say that it is "highly accurate," but this parser still falls victim to some of the same issues that most statistical parsers do, and while faster than some dependency parsers, it is not faster than all of them. The point about overfitting is valid, too, which is another reason why this "most accurate such model in the world" claim is obnoxious. It's also fair to note that their advance is in fractions of percentage points on this specific dataset over models that are 5-10 years older.
- eitally 10y agoCoincidentally, I had a parent/teacher conference with my 1st grader's teacher yesterday afternoon. Regarding reading level & comprehension, she remarked that current research indicates anything below about 98% comprehension isn't sufficient for reading "fluency". Before the past few years, the standard was 95% comprehension = fluency, but that extra few percentage points apparently make an enormous difference (probably because of colloquial & jargon edge case usages that carry specific meanings in specific contexts, but which aren't easy to programmatically detect, but that's just my supposition).
- mentat 10y agoSorry, but that just doesn't make any sense to me. Practically 70% seems like enough to understand most narrative. I've read some really difficult texts (translated German theology) and for anything of meaningful complexity 98% is unreachable without a huge vocabulary and understanding of both oddities of grammar and the construction of narrative or argument.