6 ms·
How does this relate/compare to NLTK? http://www.nltk.org/ http://www.nltk.org/
by voltagex_ 13y ago
How does this relate/compare to NLTK?
http://www.nltk.org/ http://www.nltk.org/
- andrewguenther 13y agoNLTK is a toolkit which includes a bunch of different tokenizers, parsers, taggers, stemmers, lemmatizers, and corpora. The Stanford parser which he references in this post is a parser which can be integrated to work with NLTK, but it seems the methods which he discusses are more accurate and much faster than anything included in NLTK. Also, the work he describes requires that the text already be tokenize and POS-tagged which will add some time, but not too much (and are also functions which can be performed by NLTK).
- piokuc 13y agoIt's also worth noting that the Stanford parser is a Java software, so using it from within NLTK involves calling Java from Python.
- syllogism 13y agoThe truth is nltk is basically crap for real work, but there's so little NLP software that's put proper effort into documentation that nltk still gets a lot of use. You can work your way down the vast number of nltk modules, and you'll find almost none of them are useful for real work, and those that are, ship a host of alternatives that are all much worse than the current state-of-the-art. nltk makes most sense as a teaching tool, but even then it's mostly out of date. The chapter on "Parsing" in the nltk book doesn't even really deal with statistical parsing. The dependency parsing work referenced in this post is almost all 1-3 years old, so obviously it isn't covered either. As an integration layer, nltk is so much more trouble than it's worth. You can use it to compute some of your scoring metrics, or read in a corpus, but...why bother? I'm slowly putting together an alternative, where you get exactly one tokeniser, exactly one tagger, etc. All these algorithms have the same i/o, so we shouldn't ask a user to choose one. We should just provide the best one. My previous post, on POS tagging, shows that ntlk's POS tagger is incredibly slow, and not very accurate: https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/ https://honnibal.wordpress.com/2013/09/11/a-good-part-of-spe... . nltk scores 94% in 3m56s, my 200 line implementation scores 96.8% in 12s. I used to use nltk for tokenisation and sentence-boundary detection, but this library seems better for that: https://code.google.com/p/splitta/ https://code.google.com/p/splitta/
- orf 13y agoIf that overhead comes from Python then have you tried running either of the implementations on PyPy?
- jliechti1 13y agoSince NLTK does enjoy some popularity, would it make sense to try to get some of your code into the project? I'm sure a lot of people could benefit from using/learning about more current algorithms.
- syllogism 13y agoMy code is BSD licensed, so nltk can take and adapt it in any way they choose. However, I won't do any of the work to integrate it into their class hierarchies, or to write the tests and documentation in the way that they require. I expect a successful patch request would take a lot of work. That's not a criticism; it's just how it is, given the aims, history and status of their project.
- shabadoop 13y agoI was thinking about working through the NLTK book once I'm finished with Bishop's Pattern Recognition, would you be able to recommend an alternative?
- syllogism 13y agoThere's no alternative NLP book I can really recommend, no --- sorry. It's moving too fast, the books are going out of date too quickly. You could try Hal Daume's blog, and Bob Carpenter's blog.
- sitkack 13y agoHere is a review [0] by Bob Carpenter of Foundations of Statistical Natural Language Processing [1]. [0] http://www.amazon.com/review/R2FUAZHGUOERHV http://www.amazon.com/review/R2FUAZHGUOERHV [1] http://www.amazon.com/Foundations-Statistical-Natural-Language-Processing/dp/0262133601/ref=cm_rdp_product http://www.amazon.com/Foundations-Statistical-Natural-Langua...