6 ms·

Natural Language Processing with Python, free online

255 points by motter 15y ago

drstrangevibes 15y ago
oooooh , this has just made my day!
- ObnoxiousJul 15y ago
  only the REST api is open source. I see the book as a way to sell their API for a service for which everything (algorithm and methods) are obfuscated (server side). I expected much insights for helping me on a project, I had less informations and more noise than what I already knew.
  - knowtheory 15y ago
    What on earth do you mean? The entirety of NLTK is FOSS originally produced by a university. Even if NLTK had some sort of hosted version (and I was unaware that if they do, as your post seems to claim), you could always go D/L the whole source (https://github.com/nltk/nltk https://github.com/nltk/nltk ) and use it as you see fit (it's even Apache Licensed).
  - deleted 15y ago
    [deleted]
- sebphfx 15y ago
  same here. Something I've been wanting to learn in a while being fascinated by computational linguistics.
scorpion032 15y ago
LinkGrammar: http://www.link.cs.cmu.edu/link/ http://www.link.cs.cmu.edu/link/ is the other NLP tool kit in python, may be suited for use by those who might be intimidated by the need to write their own grammar. It is integrated into and maintained by the authors of Abiword. There is a talk on this other library at Pycon, this year: https://us.pycon.org/2012/schedule/presentation/187/ https://us.pycon.org/2012/schedule/presentation/187/
- mullr 15y ago
  LinkGrammar looks very interesting and useful. But I believe that its scope is somewhat smaller than NLTK. Aside from sentence parsing, NLTK also gives you a bunch of tools for document tokenization, statistical analysis, classification, and some easily accesible corpora to try it all out on. I'd almost call it a natural language prototyping environment; it's a huge time-saver to those dabbling in the field or who need to quickly experiment with different techniques.
- daviddaviddavid 15y ago
  LinkGrammar is not really a toolkit and it is not written in Python. It is written in C.
  - scorpion032 15y ago
    Yup. My bad. There are python bindings, though.
herTTTz 15y ago
Some opinions http://news.ycombinator.com/item?id=3052540 http://news.ycombinator.com/item?id=3052540
jnbiche 15y ago
Most of NLTK runs fine in PyPy, too, resulting in significant performance enhancements.
sinzone 15y ago
There is a free NLTK cloud API: http://www.mashape.com/apis/Text-Processing http://www.mashape.com/apis/Text-Processing It includes sentiment analysis, stemming and lemmatization, part-of-speech tagging and chunking, phrase extraction and named entity recognition.
nailer 15y ago
I own the paper version of this - just a heads up, it's written primary for Language Scientists rather than developers. You may find yourself constantly pausing and looking up concepts and their meanings / practical uses on linguistics websites when returning to the book. It's a great resource, but don't expect to get started quickly.
- jnbiche 15y ago
  I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computational linguistic terminology is not gratuitous -- it's important to learn the jargon so you can look for the right type of help when you don't know how to do something. On the flip side, the Python in the book is fairly straightforward, so developers should be able to dive right in with the code. In fact, experienced developers will probably find themselves skipping over many sections that are essentially lessons on Python. Overall, this book is a good introduction to NLP and NLTK. That said, if you're at all familiar with NLP, you'd probably do just as well to dive right in and start working with the toolkit. NLTK is very well-documented and easy to work with. It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. It also includes a number of algorithms that commonly fall under ML, data mining, and IR. If you work with Python and do any kind of work with unstructured text, you need to check NLTK out.
  - nailer 15y ago
    > I'm not sure that any book on the NLTK could avoid the concepts you're referring to. Me too. I'm saying the book (like almost all technical documentation) should be task-based. Let people read the chapter with their problem - don't ask them to read a whole list of solutions to work out which one is applicable. Eg: - Title: Preparing Text for Analysis (explain what tokenizing is and why it's necessary) - Title: 'gauging opinion of a topic' (explain what sentiment analysis is) - Title 'identifying relationships' (explain logical inference) Task-based documentation doesn't seek to avoid terminology. It seeks to explain why those concepts exist and how they are useful (which encourages people to learn, as they become aware of the practical application of their knowledge).
ethanpoole 15y ago
If you're interested in natural language processing (NLP), but don't have a linguistics background, I would suggest reading Steven Pinker's The Language Instinct. It will introduce you to the necessary terminology and concepts for NLP in an easy-to-digest way. (The NLTK book has been free online for quite some time as well.)
- jnbiche 15y ago
  The Language Instinct is a great book, but unless the content of newer editions has changed significantly, it's more of an overview of linguistics in general, and language acquisition in particular. There's not much -- if any -- practical NLP. For example, looking at Amazon's statistically-improbable phrases (SIPs), I see nothing related to NLP, nor do I see any terms related to practical NLP during a quick glance of the book's index. The index includes references to a few pages on "statistics of language", but I honestly don't remember what those were about. Also, the Language Instinct was written in the early 1990s, so although new editions have been released, it's a bit dated. It's a classic read for linguists and people interested in language, but I wouldn't recommend it as an introduction to NLP.
  - ethanpoole 15y ago
    Oh no, I didn't intend for The Language Instinct as an introduction to NLP, but as a basic introduction to the fundamentals of linguistics. I mainly had in mind terms like morpheme, phoneme, scope, etc. A basic understanding of these concepts will make reading the NLTK book much easier, although it isn't necessary.
    - jnbiche 15y ago
      Ok, that makes more sense. I read your comment as being a recommendation of The Language Instinct to learn the fundamentals of NLP. But point taken, it's a good book about language.
olex 15y ago
We're using this library in class (Foundations of Language Technology) at the TU Darmstadt. From my point of view as a student, who hasn't done much Python in the past, it's pretty easy to use and works well for learning about NLP, hiding many implementation details and letting me focus on solving fairly complex tasks on data/algorithm levels.
mark_l_watson 15y ago
The book and software are both a great resource. I usually "roll my own" NLP software, but I have used NLTK for small customer text mining tasks. Definitely "batteries included." I bought the book years ago, but now, the online edition may be more current.
NnamdiJr 15y ago
Probably up there amongst the most useful Python libraries IMO. Hasn't it been available for free online for a long time now though? Anyway, in case anyone reading this missed it, the Stanford NLP class taught by Chris Manning and Dan Jurafsky starting next week (Jan 23rd) will allow programming assignments to be submitted using Python and NLTK, which is really good news. So now's a good time to get familiar with the NLTK, or for a refresher for those of us already acquainted with it.
- deleted 15y ago
  [deleted]
capttwinky 15y ago
ipython+nltk+networkx+pytables+numpy+matplotlib=full of win Seriously - text mining made fun and exploratory with open source tools. One of the text sources I like to use is the Launchpad tickets for the Ubuntu project, since they get a good amount of traffic from international end users, a professional interest of mine. It would be great to hear about some other interesting open data sets that people have found.
tikhonj 15y ago
I came across Natural Language Processing for the Working Programmer[1] recently. It's released under a creative commons license (CC-BY). It's still a work in progress, but might be interesting anyhow. [1]: http://nlpwp.org/book/ http://nlpwp.org/book/

255 points by motter 15y ago

drstrangevibes 15y ago

oooooh , this has just made my day!

ObnoxiousJul 15y ago
only the REST api is open source. I see the book as a way to sell their API for a service for which everything (algorithm and methods) are obfuscated (server side). I expected much insights for helping me on a project, I had less informations and more noise than what I already knew.
- knowtheory 15y ago
  What on earth do you mean? The entirety of NLTK is FOSS originally produced by a university. Even if NLTK had some sort of hosted version (and I was unaware that if they do, as your post seems to claim), you could always go D/L the whole source (https://github.com/nltk/nltk https://github.com/nltk/nltk ) and use it as you see fit (it's even Apache Licensed).
- deleted 15y ago
  [deleted]
sebphfx 15y ago
same here. Something I've been wanting to learn in a while being fascinated by computational linguistics.

scorpion032 15y ago

LinkGrammar: http://www.link.cs.cmu.edu/link/ http://www.link.cs.cmu.edu/link/ is the other NLP tool kit in python, may be suited for use by those who might be intimidated by the need to write their own grammar. It is integrated into and maintained by the authors of Abiword. There is a talk on this other library at Pycon, this year: https://us.pycon.org/2012/schedule/presentation/187/ https://us.pycon.org/2012/schedule/presentation/187/

mullr 15y ago
LinkGrammar looks very interesting and useful. But I believe that its scope is somewhat smaller than NLTK. Aside from sentence parsing, NLTK also gives you a bunch of tools for document tokenization, statistical analysis, classification, and some easily accesible corpora to try it all out on. I'd almost call it a natural language prototyping environment; it's a huge time-saver to those dabbling in the field or who need to quickly experiment with different techniques.
daviddaviddavid 15y ago
LinkGrammar is not really a toolkit and it is not written in Python. It is written in C.
- scorpion032 15y ago
  Yup. My bad. There are python bindings, though.

herTTTz 15y ago

Some opinions http://news.ycombinator.com/item?id=3052540 http://news.ycombinator.com/item?id=3052540

jnbiche 15y ago

Most of NLTK runs fine in PyPy, too, resulting in significant performance enhancements.

sinzone 15y ago

There is a free NLTK cloud API: http://www.mashape.com/apis/Text-Processing http://www.mashape.com/apis/Text-Processing It includes sentiment analysis, stemming and lemmatization, part-of-speech tagging and chunking, phrase extraction and named entity recognition.

nailer 15y ago

I own the paper version of this - just a heads up, it's written primary for Language Scientists rather than developers. You may find yourself constantly pausing and looking up concepts and their meanings / practical uses on linguistics websites when returning to the book. It's a great resource, but don't expect to get started quickly.

jnbiche 15y ago
I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computational linguistic terminology is not gratuitous -- it's important to learn the jargon so you can look for the right type of help when you don't know how to do something. On the flip side, the Python in the book is fairly straightforward, so developers should be able to dive right in with the code. In fact, experienced developers will probably find themselves skipping over many sections that are essentially lessons on Python. Overall, this book is a good introduction to NLP and NLTK. That said, if you're at all familiar with NLP, you'd probably do just as well to dive right in and start working with the toolkit. NLTK is very well-documented and easy to work with. It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. It also includes a number of algorithms that commonly fall under ML, data mining, and IR. If you work with Python and do any kind of work with unstructured text, you need to check NLTK out.
- nailer 15y ago
  > I'm not sure that any book on the NLTK could avoid the concepts you're referring to. Me too. I'm saying the book (like almost all technical documentation) should be task-based. Let people read the chapter with their problem - don't ask them to read a whole list of solutions to work out which one is applicable. Eg: - Title: Preparing Text for Analysis (explain what tokenizing is and why it's necessary) - Title: 'gauging opinion of a topic' (explain what sentiment analysis is) - Title 'identifying relationships' (explain logical inference) Task-based documentation doesn't seek to avoid terminology. It seeks to explain why those concepts exist and how they are useful (which encourages people to learn, as they become aware of the practical application of their knowledge).

ethanpoole 15y ago

If you're interested in natural language processing (NLP), but don't have a linguistics background, I would suggest reading Steven Pinker's The Language Instinct. It will introduce you to the necessary terminology and concepts for NLP in an easy-to-digest way. (The NLTK book has been free online for quite some time as well.)

jnbiche 15y ago
The Language Instinct is a great book, but unless the content of newer editions has changed significantly, it's more of an overview of linguistics in general, and language acquisition in particular. There's not much -- if any -- practical NLP. For example, looking at Amazon's statistically-improbable phrases (SIPs), I see nothing related to NLP, nor do I see any terms related to practical NLP during a quick glance of the book's index. The index includes references to a few pages on "statistics of language", but I honestly don't remember what those were about. Also, the Language Instinct was written in the early 1990s, so although new editions have been released, it's a bit dated. It's a classic read for linguists and people interested in language, but I wouldn't recommend it as an introduction to NLP.
- ethanpoole 15y ago
  Oh no, I didn't intend for The Language Instinct as an introduction to NLP, but as a basic introduction to the fundamentals of linguistics. I mainly had in mind terms like morpheme, phoneme, scope, etc. A basic understanding of these concepts will make reading the NLTK book much easier, although it isn't necessary.
  - jnbiche 15y ago
    Ok, that makes more sense. I read your comment as being a recommendation of The Language Instinct to learn the fundamentals of NLP. But point taken, it's a good book about language.

olex 15y ago

We're using this library in class (Foundations of Language Technology) at the TU Darmstadt. From my point of view as a student, who hasn't done much Python in the past, it's pretty easy to use and works well for learning about NLP, hiding many implementation details and letting me focus on solving fairly complex tasks on data/algorithm levels.

mark_l_watson 15y ago

The book and software are both a great resource. I usually "roll my own" NLP software, but I have used NLTK for small customer text mining tasks. Definitely "batteries included." I bought the book years ago, but now, the online edition may be more current.

NnamdiJr 15y ago

Probably up there amongst the most useful Python libraries IMO. Hasn't it been available for free online for a long time now though? Anyway, in case anyone reading this missed it, the Stanford NLP class taught by Chris Manning and Dan Jurafsky starting next week (Jan 23rd) will allow programming assignments to be submitted using Python and NLTK, which is really good news. So now's a good time to get familiar with the NLTK, or for a refresher for those of us already acquainted with it.

deleted 15y ago
[deleted]

capttwinky 15y ago

ipython+nltk+networkx+pytables+numpy+matplotlib=full of win Seriously - text mining made fun and exploratory with open source tools. One of the text sources I like to use is the Launchpad tickets for the Ubuntu project, since they get a good amount of traffic from international end users, a professional interest of mine. It would be great to hear about some other interesting open data sets that people have found.

tikhonj 15y ago

I came across Natural Language Processing for the Working Programmer[1] recently. It's released under a creative commons license (CC-BY). It's still a work in progress, but might be interesting anyhow. [1]: http://nlpwp.org/book/ http://nlpwp.org/book/