4 ms·
Anyone familiar enough with this library to explain how they've achieved such impressive performance? Otherwise, may dig into it for a blog post when I have tim
by avinium 7y ago
Anyone familiar enough with this library to explain how they've achieved such impressive performance? Otherwise, may dig into it for a blog post when I have time.
- lettergram 7y agoOne advantage is the core appears to be written in C/C++. NLTK is python and SpaCy is a lot of C/C++ with a significant amount of python. I’ll have to dig in further, but I suspect there’s a lot of optimization for the task at hand here. SpaCy and NLTK tries to be much more general for NLP.
- avinium 7y agoI was more so referring to spacy, where (IIRC) a lot of the lower-level tokenization & parsing is written in Cython, so (I assume) the simple fact that Bling is written in C++ wouldn't explain the huge performance gains.
- stochastic_monk 7y agoThere’s still a good amount of overhead in Cython portions of code unless there are no python objects or GIL involved, and then there’s still overhead communicating between C and Python worlds. That’s part of why I stopped writing Cython and just use pybind11.
- sergeio76 7y agoThere are deterministic finite state machines underneath implemented in lean C++. These automata allow to implement operations with strings optimally or close to that. In the readme file there is a link to how to recompile linguistic resources, if you look inside the makefile that is used for resource compilation you will the steps.
- mlacks 7y agoWhat is lean C++?
- Abishek_Muthian 7y agoI think he just means efficient C++ programming.
- taf2 7y ago#define WIN32_LEAN_AND_MEAN Maybe ?
- avinium 7y agoThanks - will dive in and start checking things out.
- criddell 7y agoDo you use any kind of library or typical design pattern for your FSM implementation? When I've used them, I typically always go back to very C-like code (enum for the states, a state variable, and a big switch statement inside a loop).
- sergeio76 7y agoWe don't represent automata as a code, that leads to compiler errors etc., we represent them as graphs, just as data. Inside the code we use: state --> int set of states --> sorted / unique'd array of int's input symbol --> int output symbol --> int transition function is abstracted out behind an interface and implemented differently based on whether automaton is read-only or changeable and based on the state etc.
- criddell 7y agoI'm going to have to think about that a bit. Maybe I'm missing the point, but it sounds like an interpreter. Thanks for posting the code. This feels like something I need to understand.