7 ms·
Natural Language Processing for the Working Programmer
- nubela 14y agoI was working with NLP, and its various toolkits (python-nltk I'm looking at you). The thing about NLP is, there just isn't enough libraries (for humans) to simply plug NLP into use. Even nltk, the premier python library for NLP, seems to be an NLP core-library for building NLP solutions, rather than for building NLP-powered apps. It also seems to extremely unpythonic. Is there a missing link there? I don't know. So I took a few days and built an extremely simple NER (named entity recognition) engine and made it extremely easy for any programmer to begin using. See http://blog.nerily.com/howto-train-your-own-modelset-for-your-custom http://blog.nerily.com/howto-train-your-own-modelset-for-you... . We'll see how NLP becomes more easily accessible with better tools to come over time.
- bane 14y agoyeah, for the vast majority of uses, most people really want to do just a fairly small set of things fairly well. NER comes to mind, lots and lots and lots of toolkits for building up to NER, but very few that let submit English text and get back a list of people, places and things without having to virtually build my own NER system from scratch anyways. Give me NER, Entity relationships (ER) and a couple kinds of sentiment analysis scoring (SA) (which can be jump started with decent NER) and I've pretty much exhausted 95% of what I'd ever want to do. I really really really don't need yet another library to do sentence tokenization, term tokenization, tf counting and stemming. If I was building a free text indexer or bayesian filter or some such it might be useful, but I'm probably not, there are far better solutions to those domain than I'm likely to come up with, but there aren't for NER, ER and SA.
- agibsonccc 14y agoI can't recommend any libraries "for humans" for this, there are APIs out there for it. The main problem with many NLP libs (and data mining applications in general) has a lot to do with how much memory good models can take up when doing it in order for it to be accurate at all. Here are a few APIs and libs that might be useful though: (Disclaimer: publisher of this one here) https://www.mashape.com/agibsonccc/semantic-analytic https://www.mashape.com/agibsonccc/semantic-analytic There are other text processing APIs on there as well. As for libraries, I primarily come from the JVM camp for NLP, but I would recommend the following libraries: http://nlp.stanford.edu/software/index.shtml http://nlp.stanford.edu/software/index.shtml (Comprehensive) http://code.google.com/p/clearnlp/ http://code.google.com/p/clearnlp/ (Fairly simple) My favorite is cleartk (http://code.google.com/p/cleartk/ http://code.google.com/p/cleartk/ ) mainly due to the fact it's a consistent interface, but UIMA itself can be a difficult toolchain to pick up, and I could understand most of these being overkill for many simple applications people may have in mind.
- bane 14y agoI've heard some good things about OpenNLP as well http://opennlp.apache.org/ http://opennlp.apache.org/ but haven't had the time to look at it with any detail.
- agibsonccc 14y agoOpenNLP is great. I've used it for a lot of subtasks, but nothing that produces end results as described earlier. It's an amazing library for building NLP systems, but doesn't produce anything directly (named entity recognition, etc) Typically it's coupled with other libraries. The big problem I think with NLP in general is typically to do anything, you need a pipeline. (Sentence segmentation, tokenization, part of speech tagging) usually at a bare minimum. Then from there you can do named entity recognition or other tasks that produce actual usable results.
- saryant 14y agoI've used the Stanford NLP library extensively for NER. I made heavy use of it in my senior thesis project. It's pretty straightforward to use their library to read a document and output an XML file containing NER data (and lots of other fun stuff). For instance, from the sentence: > World War II, or the Second World War (often abbreviated as WWII or WW2), was a global military conflict lasting from 1939 to 1945, which involved most of the world's nations, including all of the great powers, eventually forming two opposing military alliances, the Allies and Axis. Stanford NLP NER will output the following entities: "World War II" - MISC "Second World War" - MISC "1939 to 1945" - DATE - NORMALIZED 1939/1945 "Axis" - MISC You can view the output of Stanford's CoreNLP library (NER + dependency grammar + coreference resolution + some other stuff) for the Wikipedia article on World War II in my github repo: https://raw.github.com/ryantanner/thesis/master/data/ww2sample.txt.xml https://raw.github.com/ryantanner/thesis/master/data/ww2samp... edit: I should add that the real fun (for me) came from combining NER with dependency grammars and coreference resolution. It makes it very easy to turn Stanford NLP's output into a knowledge graph combining a large number of documents.
- agibsonccc 14y agoFor those who want to play around with dynamic output: http://nlp.stanford.edu:8080/parser/ http://nlp.stanford.edu:8080/parser/ This is a bit more human friendly.
- nubela 14y agohttp://www.nerily.com/#demo http://www.nerily.com/#demo -> Or this with a JSON API output.
- ninjin 14y agoHere is the whole Stanford CoreNLP suite with visualisation output: http://nlp.stanford.edu:8080/corenlp/ http://nlp.stanford.edu:8080/corenlp/ It helps me greatly when it comes to interpreting the dependency structure. If you need an example sentence: "Stanford University is located in California. It is a great university." I also know that Microsoft Research has a demo online of their NLP tools: http://msrsplatdemo.cloudapp.net/ http://msrsplatdemo.cloudapp.net/ (Silverlight required) I don't think you can download the tools though, but they do offer to provide you with an API token to call their service from their cloud. Potential conflict of interest: I wrote parts of the CoreNLP visualiser. Edit: Added example sentence.
- PeterisP 14y agoWell, if you want to get a really good accuracy, then you need a model specific to your domain - which most likely you'll need to train up yourself. The NLP systems leading in competitions such as CONLL conference tend to be publicly available, so you can get a "general purpose" system there; but the current way usually is to train specific model for a specific purpose - since if you don't have a predetermined purpose, you can't really tell which of items should be tagged as places (instead of things); you can't tell which things should be tagged as 'things' and in what way they should be classified deeper - the list of classes tends to be application-specific.
- eric_bullington 14y agoWhile it's possible that some parts of NLTK may be unpythonic, I disagree that it's somehow inaccessible or hard to use. The NLTK project is fairly well-documented, and I've never had any difficulty using it. Also, all the NLTK code I've ever examined looks Pythonic to me, unlike the Java-style Python that I sometimes see coming out of academia; they follow PEP8 and have a developer style guide. There may be pockets of unpythonic code, but I've not come across any. NLTK is also free and open-source, with a liberal license (Apache), which I appreciate greatly. Also, I don't understand what you mean by nltk being a library for building NLP solutions, rather than NLP-powered apps. Can you expand on that? Re: NER, I found this gist (not my own) for a basic example of entity extraction with nltk: https://gist.github.com/322906/90dea659c04570757cccf0ce1e6d26c9d06f9283 https://gist.github.com/322906/90dea659c04570757cccf0ce1e6d2... This looks pretty straight-forward to me. What NLP toolkit are you using for the NER service your Chrome extension calls, if not NLTK?
- nubela 14y agoIt isn't just extraction, but the entire process, from training a dictionary, to training it, etc. Have you tried training your own dictionary to extract custom entities? It is "non-trivial".
- hnriot 14y agoNERILY appears to be closed source and only usable on a subscription pricing model. It appears to be similar to zemanta.com - unless I am mistaken and this is also available as a library?
- nubela 14y agoIt is indeed closed source and available as a NLP as a Service model, what we do have instead is a quick-to-deploy ReST API, with JSON output, and even a chrome extension to train your own dictionary.
- superbobry 14y agoI wonder if PDF or Epub version is available? couldn't find it on the site.
- Ellos 14y agoHere you go http://nlpwp.org/nlpwp.pdf http://nlpwp.org/nlpwp.pdf
- superbobry 14y agoGreat, thank you!
- danso 14y agoI'm not that familiar with Haskell and the past week's HN frontpage articles on Monads was just confusing...but what is it about Haskell that makes it more useful for NLP than, say, Python?
- yummyfajitas 14y agoNot much. It's a more expressive and cleaner language, but on the other hand python has NLTK + scipy community. Scala (or Java) is another great NLP language. It's got decent libraries (openNLP, mallet, mahout), hadoop, and Scala is almost as nice as Haskell.
- robertfw 14y agoBy extension, Clojure can use those same java libraries
- KirinDave 14y ago> Not much. It's a more expressive and cleaner language, but on the other hand python has NLTK + scipy community. Haskell's mechanisms for defining parsers, lexers, and other pattern match tools is so good it probably passes over the line from "pretty" to "objectively better". A lot of people who need to lex and parse data and then act on it turn to Haskell. It has some really remarkable and efficient libraries. And even for "common" target languages it's reasonable to write extremely fast parsers. With tuning, projects like Aeson are among some of the fastest JSON parsers and writers out there (only a few projects exceed its speed and resource efficiency in ANY runtime).
- ceautery 14y agoI think promoting this should have waited until the section on Natural Language Processing said more than "stub".
- mark_l_watson 14y agoNot open source, but I provide a free web service endpoint for my NLP system: http://kbsportal.com/ http://kbsportal.com/ Use the form demo page to use it interactively.
- MojoJolo 14y agoI'm always fascinated with NLP. My undergrad works around it. And currently doing a research for my MS degree. It's about a variant of automatic summarization, wherein I extract the most important sentences in an article. I'll open an API for it soon. :) If you're interested, just contact me in my email (check my profile for it). In the meantime, here's a preview of what it can do. http://goo.gl/Lz7Vr http://goo.gl/Lz7Vr
- pdat 14y agoMaluuba provides an API for natural language processing: http://dev.maluuba.com/ http://dev.maluuba.com/
- danieldk 14y agoOne of the authors here: we wrote this during the Pragmatic Programmer's writing month in 2010 and some more in 2011. Then I got caught up writing my PhD thesis, and now a new job (as an NLP engineer, but in Java ;)). So, the book is basically frozen. We hope to have more time in the future to continue the writing...
- phantom-scald 14y agoNice endeavor, but finished up as the most endeavors - unfinished. :) That was the first book in NLP (and the only for now) that I read. I've been interested both in NLP and Haskell. In that respect it fitted, thanks! A few points to criticize. For the frequency list one should use multisets, not dictionaries. There are a few multiset packages at Hackage. Suffix arrays are badly explained. Monads - very badly. With tagging there was an impression that it could be explained simpler. Many things are announced but not touched. The book is not a book in fact, it's more like an article. Perhaps reconsider it in that way? But oke, hopefully you will find time to continue it as a book. Perhaps meanwhile you can recommend some other book to continue reading on NLP?
- danieldk 14y agoSpeech and Language Processing by Jurafsky and Martin Foundations of Statistical Natural Language Processing by Manning and Schütze I have to say comments like yours are not really encouraging to continue writing ;).
- phantom-scald 14y agoI'm sorry for that! All other sections were written nicely or OK, and I appreciate for what I picked up from the book. I just wanted to point out some places needed to be reworked in case you continue. Myself being in industry, I know how hard, near to impossible it is to find time for anything extra than work and family. And a decent book requires approximately the same amount of effort as finishing PhD. Perhaps that was my frustration coming out of the projects I had to abandon. :( Thanks for refs!
- 14y ago
- ExpiredLink 14y ago> 1.2. What is natural language processing? > Stub
- rlf 14y agoWould be nice if there was a link to the TOC in the header of each page. Otherwise, very useful.
- myth_drannon 14y agoJust FYI : there is a NLP class starting soon on Coursera https://class.coursera.org/nlangp-001 https://class.coursera.org/nlangp-001