4 ms·
Note that the top level summary shows it is 10x faster than Spacey, which is a more impressive achievement than being 10x faster than NLTK (it is 20x faster).
by rlayton2 8y ago
Note that the top level summary shows it is 10x faster than Spacey, which is a more impressive achievement than being 10x faster than NLTK (it is 20x faster).
NLTK is designed for learning, not for production systems, speed or efficiency. The code is often the most straight-forward way to implement an algorithm, especially to read the code later.
- deleted 8y ago[deleted]
- linkingday 8y agoTo further your point, NLTK is written in pure Python whereas the backend for spaCy, thinc, is written in C/C++/Python. Really no comparison here.
- avinium 8y agoAnyone familiar enough with this library to explain how they've achieved such impressive performance? Otherwise, may dig into it for a blog post when I have time.
- lettergram 8y agoOne advantage is the core appears to be written in C/C++. NLTK is python and SpaCy is a lot of C/C++ with a significant amount of python. I’ll have to dig in further, but I suspect there’s a lot of optimization for the task at hand here. SpaCy and NLTK tries to be much more general for NLP.
- avinium 8y agoI was more so referring to spacy, where (IIRC) a lot of the lower-level tokenization & parsing is written in Cython, so (I assume) the simple fact that Bling is written in C++ wouldn't explain the huge performance gains.
- stochastic_monk 8y agoThere’s still a good amount of overhead in Cython portions of code unless there are no python objects or GIL involved, and then there’s still overhead communicating between C and Python worlds. That’s part of why I stopped writing Cython and just use pybind11.
- sergeio76 8y agoThere are deterministic finite state machines underneath implemented in lean C++. These automata allow to implement operations with strings optimally or close to that. In the readme file there is a link to how to recompile linguistic resources, if you look inside the makefile that is used for resource compilation you will the steps.
- mlacks 8y agoWhat is lean C++?
- Abishek_Muthian 8y agoI think he just means efficient C++ programming.
- taf2 8y ago#define WIN32_LEAN_AND_MEAN Maybe ?
- avinium 8y agoThanks - will dive in and start checking things out.
- criddell 8y agoDo you use any kind of library or typical design pattern for your FSM implementation? When I've used them, I typically always go back to very C-like code (enum for the states, a state variable, and a big switch statement inside a loop).
- sergeio76 8y agoWe don't represent automata as a code, that leads to compiler errors etc., we represent them as graphs, just as data. Inside the code we use: state --> int set of states --> sorted / unique'd array of int's input symbol --> int output symbol --> int transition function is abstracted out behind an interface and implemented differently based on whether automaton is read-only or changeable and based on the state etc.
- rthy 8y agoI have been experimenting with tokenization in Rust in https://github.com/rth/vtext https://github.com/rth/vtext, mostly relying on Unicode segmentation with a few custom rules for tokenization. The obtained performance was also around 10x faster than spacy for comparable precision (see benchmarks section of the readme).