4 ms·

Natural Language Processing in Python

107 points by fheisler 12y ago

andrewguenther 12y ago
NLTK is a wonderful toolkit. Its selection of corpera is great and its many utility functions for processing text are incredibly useful and easily extendable. That being said, a lot of the ML, porter, and stemmer implementations are a bit out of date from the current cutting edge in the field. If you are interested in using NLTK for serious projects, I highly recommend writing custom implementations of these modules or using other libraries.
- rajibsingh 12y ago
  which other libraries would you recommend?
  - andreasvc 12y ago
    You could look at pattern and TextBlob. For machine learning turn to scikit-learn.
  - nyir 12y ago
    There are also Python wrappers for Stanford CoreNLP.
- andreasvc 12y ago
  It should be noted that NLTK is a library intended for didactic purposes. Efficient implementation and/or state-of-the-art algorithms are not priorities.
- acosmism 12y ago
  Agreed. I too had given a tutorial on this a while back (https://www.youtube.com/watch?v=kKe4M4iSclc https://www.youtube.com/watch?v=kKe4M4iSclc) and nltk is a quick way to prototype up something neat,but yes- if you need more than "toy" functionality, there are currently better tools for the job.
  - bane 12y ago
    Thanks for the talk, I enjoyed the video. Do you have any good pointers to building a named entity extractor with NLTK?
    - acosmism 12y ago
      Thank you, I'm glad you you enjoyed it - I hope to have one on more advanced topics at some point. If you are looking for a named entity extractor sample, I have a sample from my talk on github: https://github.com/shanbady/NLTK-Boston-Python-Meetup/blob/master/named_ent_chunker.py https://github.com/shanbady/NLTK-Boston-Python-Meetup/blob/m... The sample uses the built-in named entity tagger but nltk also has support for leveraging the Stanford named entity tagger: http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanford http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanfo...
      - bane 12y ago
        Thanks for the links. Please put me on the list for when a video of your second talk is out.
  - ehurrell 12y ago
    Seconding the power of NLTK for prototyping, it's very easy to build something quickly to test an approach on your data, and at least in my experience, designed in such a way that I could easily replace elements with my own code.

107 points by fheisler 12y ago

NLTK is a wonderful toolkit. Its selection of corpera is great and its many utility functions for processing text are incredibly useful and easily extendable. That being said, a lot of the ML, porter, and stemmer implementations are a bit out of date from the current cutting edge in the field. If you are interested in using NLTK for serious projects, I highly recommend writing custom implementations of these modules or using other libraries.

rajibsingh 12y ago
which other libraries would you recommend?
- andreasvc 12y ago
  You could look at pattern and TextBlob. For machine learning turn to scikit-learn.
- nyir 12y ago
  There are also Python wrappers for Stanford CoreNLP.
andreasvc 12y ago
It should be noted that NLTK is a library intended for didactic purposes. Efficient implementation and/or state-of-the-art algorithms are not priorities.
acosmism 12y ago
Agreed. I too had given a tutorial on this a while back (https://www.youtube.com/watch?v=kKe4M4iSclc https://www.youtube.com/watch?v=kKe4M4iSclc) and nltk is a quick way to prototype up something neat,but yes- if you need more than "toy" functionality, there are currently better tools for the job.
- bane 12y ago
  Thanks for the talk, I enjoyed the video. Do you have any good pointers to building a named entity extractor with NLTK?
  - acosmism 12y ago
    Thank you, I'm glad you you enjoyed it - I hope to have one on more advanced topics at some point. If you are looking for a named entity extractor sample, I have a sample from my talk on github: https://github.com/shanbady/NLTK-Boston-Python-Meetup/blob/master/named_ent_chunker.py https://github.com/shanbady/NLTK-Boston-Python-Meetup/blob/m... The sample uses the built-in named entity tagger but nltk also has support for leveraging the Stanford named entity tagger: http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanford http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanfo...
    - bane 12y ago
      Thanks for the links. Please put me on the list for when a video of your second talk is out.
- ehurrell 12y ago
  Seconding the power of NLTK for prototyping, it's very easy to build something quickly to test an approach on your data, and at least in my experience, designed in such a way that I could easily replace elements with my own code.