3 ms·

Very cool! I'm familiar with NLTK, but not SpaCy. It looks like the speed benefits of SpaCy could help greatly in processing large volumes of unstructured text.

by grej 10y ago

Very cool! I'm familiar with NLTK, but not SpaCy. It looks like the speed benefits of SpaCy could help greatly in processing large volumes of unstructured text. I'm curious what type of feedback you've gotten from users that have migrated from NLTK to SpaCy.

Smerity 10y ago
I'm not the author but can speak to some of this. Some amount of this was answered by the author himself in his blog post "Dead Code Should Be Buried – Why I Didn't Contribute to NLTK"[1]. While there are portions on NLTK that are useful, it suffers from having too many things under the hood and a lack of consistent maintenance. Many of them are superglued external libraries while others have not been maintained or have never been made to work well with other parts of NLTK. The biggest example of this was that, for some time, it wasn't certain how the NLTK part of speech tagger was trained[2]. While that has been remedied, the remedy uses part of speech tagging code inspired via earlier work by the author of spaCy[3][4]. spaCy has been made from the ground up to support more modern deep learning features (easy integration of Keras is quite cool but I'm actually meaning the default inclusion of word vectors which has existed since early spaCy) while having a consistent and clean API that covers many sane use cases. The API is highly Pythonic but the underlying performance critical components are written in Cython, meaning very few parsers are actually able to beat spaCy in speed (especially when using CPU and not GPU). Regarding speed, multi-threading is also a relatively trivial process - Cython releases the GIL around the syntactic parser. As a reference to speed, they used it to tag and parse every Reddit comment from 2015 to then generate "sense2vec"[5] at over 100,000 words per second using 4 threads. Finally, and most importantly, spaCy compares to and remains competitive with state of the art results and all of it remains under an MIT license. As you might have guessed, I'm a fan :) Note: I was in the same NLP lab as Matthew Honnibal for my undergraduate so will admit positive bias ;) This was long before spaCy was a thing however - he was simply a fascinating and interesting guy for other reasons ^_^ [1]: https://explosion.ai/blog/dead-code-should-be-buried https://explosion.ai/blog/dead-code-should-be-buried [2]: https://github.com/nltk/nltk/issues/1063 https://github.com/nltk/nltk/issues/1063 [3]: https://github.com/nltk/nltk/pull/1143 https://github.com/nltk/nltk/pull/1143 [4]: https://explosion.ai/blog/part-of-speech-pos-tagger-in-python https://explosion.ai/blog/part-of-speech-pos-tagger-in-pytho... [5]: https://explosion.ai/blog/sense2vec-with-spacy https://explosion.ai/blog/sense2vec-with-spacy
madenine 10y ago
As someone else in the thread said, spaCy is my 95% solution; its what I use first to get me going. The biggest benefit you'll see right off the bat is speed. SpaCy is much faster than most other NLP tools out there.

by grej 10y ago

Smerity 10y ago

I'm not the author but can speak to some of this. Some amount of this was answered by the author himself in his blog post "Dead Code Should Be Buried – Why I Didn't Contribute to NLTK"[1]. While there are portions on NLTK that are useful, it suffers from having too many things under the hood and a lack of consistent maintenance. Many of them are superglued external libraries while others have not been maintained or have never been made to work well with other parts of NLTK. The biggest example of this was that, for some time, it wasn't certain how the NLTK part of speech tagger was trained[2]. While that has been remedied, the remedy uses part of speech tagging code inspired via earlier work by the author of spaCy[3][4]. spaCy has been made from the ground up to support more modern deep learning features (easy integration of Keras is quite cool but I'm actually meaning the default inclusion of word vectors which has existed since early spaCy) while having a consistent and clean API that covers many sane use cases. The API is highly Pythonic but the underlying performance critical components are written in Cython, meaning very few parsers are actually able to beat spaCy in speed (especially when using CPU and not GPU). Regarding speed, multi-threading is also a relatively trivial process - Cython releases the GIL around the syntactic parser. As a reference to speed, they used it to tag and parse every Reddit comment from 2015 to then generate "sense2vec"[5] at over 100,000 words per second using 4 threads. Finally, and most importantly, spaCy compares to and remains competitive with state of the art results and all of it remains under an MIT license. As you might have guessed, I'm a fan :) Note: I was in the same NLP lab as Matthew Honnibal for my undergraduate so will admit positive bias ;) This was long before spaCy was a thing however - he was simply a fascinating and interesting guy for other reasons ^_^ [1]: https://explosion.ai/blog/dead-code-should-be-buried https://explosion.ai/blog/dead-code-should-be-buried [2]: https://github.com/nltk/nltk/issues/1063 https://github.com/nltk/nltk/issues/1063 [3]: https://github.com/nltk/nltk/pull/1143 https://github.com/nltk/nltk/pull/1143 [4]: https://explosion.ai/blog/part-of-speech-pos-tagger-in-python https://explosion.ai/blog/part-of-speech-pos-tagger-in-pytho... [5]: https://explosion.ai/blog/sense2vec-with-spacy https://explosion.ai/blog/sense2vec-with-spacy

madenine 10y ago

As someone else in the thread said, spaCy is my 95% solution; its what I use first to get me going. The biggest benefit you'll see right off the bat is speed. SpaCy is much faster than most other NLP tools out there.