5 ms·

Does anyone familiar with NLP know how the sentence trees are parsed? Can something like python's nltk do that? I always assumed that was very difficult to do w

by bhelx 13y ago

Does anyone familiar with NLP know how the sentence trees are parsed? Can something like python's nltk do that? I always assumed that was very difficult to do well.

miket 13y ago
The sentence trees are parsed with the Stanford NLP Parser, available here: http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml
- bhelx 13y ago
  Cool. Thank you!
Cyranix 13y ago
Some out-of-the-box parsing can be done with NLTK and OpenNLP. Just be mindful of the fact that the quality of the parsing is fairly dependent on whether the corpora you provide are sufficiently similar to the corpora that the models (part-of-speech, parsing, chunking, etc.) were trained on.
ninjin 13y ago
Parsing is one of the most competitive areas of research in NLP and yes, you are correct, doing it well is very difficult. State-of-the-art performance on newswire data is around 95% if you score local structure. miket has already replied with a well-established lexical parser. The Stanford parser is one of the most robust out there, you could also go for the BLLIP (or Charniak or Charniak and Johnson) re-ranking parser [1] that although a bit old and fiddly (the GitHub version is very solid though) tends to perform well. However, both the Stanford and BLLIP parsers are constituency parsers (now we are going to go all linguistic here) and there is an alternate paradigm of dependency grammars [2]. I have heard a couple of times that Google is running a variant of the MaltParser [3] in-house and it is a very solid piece of work (the command-line interface requires some patience though, but there is plenty of documentation). People for example in Information Extraction (IE) tends to favour dependency grammars, probably since you get relevant modifiers closer to your point of interest in the tree. Now, we could also go into richer grammars like Combinatory Categorial Grammar (CCG) [4] and Head-driven Phrase Structure Grammar (HPSG) [5] but I think this reply is long enough by now... Personally I favour dependency parsing since there is more annotated data for it across plenty of languages. Also, the paradigm can cope with a lot of linguistic wonkiness that is very much present in language other than English (free word order and non-projective structures for example) [1]: https://github.com/dmcc/bllip-parser https://github.com/dmcc/bllip-parser [2]: http://en.wikipedia.org/wiki/Dependency_grammar http://en.wikipedia.org/wiki/Dependency_grammar [3]: http://maltparser.org/ http://maltparser.org/ [4]: http://en.wikipedia.org/wiki/Combinatory_categorial_grammar http://en.wikipedia.org/wiki/Combinatory_categorial_grammar [5]: http://en.wikipedia.org/wiki/Head-driven_phrase_structure_grammar http://en.wikipedia.org/wiki/Head-driven_phrase_structure_gr...
- bhelx 13y ago
  thank you ninjin

by bhelx 13y ago

Does anyone familiar with NLP know how the sentence trees are parsed? Can something like python's nltk do that? I always assumed that was very difficult to do well.

miket 13y ago

The sentence trees are parsed with the Stanford NLP Parser, available here: http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml

bhelx 13y ago
Cool. Thank you!

Cyranix 13y ago

Some out-of-the-box parsing can be done with NLTK and OpenNLP. Just be mindful of the fact that the quality of the parsing is fairly dependent on whether the corpora you provide are sufficiently similar to the corpora that the models (part-of-speech, parsing, chunking, etc.) were trained on.

ninjin 13y ago

Parsing is one of the most competitive areas of research in NLP and yes, you are correct, doing it well is very difficult. State-of-the-art performance on newswire data is around 95% if you score local structure. miket has already replied with a well-established lexical parser. The Stanford parser is one of the most robust out there, you could also go for the BLLIP (or Charniak or Charniak and Johnson) re-ranking parser [1] that although a bit old and fiddly (the GitHub version is very solid though) tends to perform well. However, both the Stanford and BLLIP parsers are constituency parsers (now we are going to go all linguistic here) and there is an alternate paradigm of dependency grammars [2]. I have heard a couple of times that Google is running a variant of the MaltParser [3] in-house and it is a very solid piece of work (the command-line interface requires some patience though, but there is plenty of documentation). People for example in Information Extraction (IE) tends to favour dependency grammars, probably since you get relevant modifiers closer to your point of interest in the tree. Now, we could also go into richer grammars like Combinatory Categorial Grammar (CCG) [4] and Head-driven Phrase Structure Grammar (HPSG) [5] but I think this reply is long enough by now... Personally I favour dependency parsing since there is more annotated data for it across plenty of languages. Also, the paradigm can cope with a lot of linguistic wonkiness that is very much present in language other than English (free word order and non-projective structures for example) [1]: https://github.com/dmcc/bllip-parser https://github.com/dmcc/bllip-parser [2]: http://en.wikipedia.org/wiki/Dependency_grammar http://en.wikipedia.org/wiki/Dependency_grammar [3]: http://maltparser.org/ http://maltparser.org/ [4]: http://en.wikipedia.org/wiki/Combinatory_categorial_grammar http://en.wikipedia.org/wiki/Combinatory_categorial_grammar [5]: http://en.wikipedia.org/wiki/Head-driven_phrase_structure_grammar http://en.wikipedia.org/wiki/Head-driven_phrase_structure_gr...

bhelx 13y ago
thank you ninjin