sergeio76

4 ms·

1.
by sergeio76 7y ago
Mainly it will help improve latency at inference time. For preprocessing training one can always add more threads, but not at inference time.
2.
by sergeio76 7y ago
State of the art performance for BERT :-)
3.
by sergeio76 7y ago
We don't represent automata as a code, that leads to compiler errors etc., we represent them as graphs, just as data. Inside the code we use: state --> int set of states --> sorted / unique'd array of int's input s
4.
by sergeio76 7y ago
fefe23, Sorry we did not put all this information into the GitHub readme, we will put more documentation into the doc folder soon, I hope it will answer some of your questions. To specifically answer your questions: Regular expressions are
5.
by sergeio76 7y ago
MWE: 1. given your corpus you run something like word2phrase algorithm on it. 2. Compute the phrases. 3. Compile them as one Min DFA and do left-most-longest match between the input sequence of tokens and the compiled DFA. We will include
6.
by sergeio76 7y ago
Maybe you are right from theoretical point of view, however 1.latency at inference time may be an overkill, 2.look at success of fasttext model from Facebook, it computes vectors for words and ngrams! as well. What you will have to learn wi
7.
by sergeio76 7y ago
The library we have published is a finite state machines manipulation library first of all, also it is developed to support linguistic applications (large number of entries, unicode, compile once use many times, etc.) Tokenization is one of
8.
by sergeio76 7y ago
There are deterministic finite state machines underneath implemented in lean C++. These automata allow to implement operations with strings optimally or close to that. In the readme file there is a link to how to recompile linguistic resour
9.
by sergeio76 7y ago
No it does not, we are thinking about it. Extra segmentation of tokens is a great idea, I only don't like (who cares? :-)) inconsistencies that BPE sometimes produce. Looking at how BERT tokenizers for example: "tranformers"
10.
by sergeio76 7y ago
Thank you for remembering us ;)