3 ms·
Two things about this baffle me. 1. Why only put perf data in the blurb? To me it would be much more helpful to first see a few API calls this allows that solv
by fefe23 7y ago
Two things about this baffle me.
1. Why only put perf data in the blurb? To me it would be much more helpful to first see a few API calls this allows that solve problems I have had before, or may have in the future. It remains mostly opaque to me what this software does.
2. If this is mostly about splitting text into words, why is the code dump dozens of directories with many hundreds of source code files in there?
I understand that they are proud of their perf numbers, but those are (to me) not as important as first understanding what I could do with the project.
Regular expression mangling sounds like it could be useful for much more than word splitting in a search engine context.
- sergeio76 7y agoThe library we have published is a finite state machines manipulation library first of all, also it is developed to support linguistic applications (large number of entries, unicode, compile once use many times, etc.) Tokenization is one of its applications. We needed to start somewhere. In our Deep Learning Era not everything that we have created is relevant, but tokenization is. What we might add to the project is BPE or BPE- variant, support East Asian languages, Multi Word Expressions ("New York" as one token).
- fefe23 7y agoCould you please explain what kind of manipulation your library supports on finite state machines? Does it contain a regex engine to easily create the state machines in the first place? Does it have a JIT? Do the state machines operate on char? wchar_t? These are the kinds of details that I would love to see on the github entry page. It's obvious to you what your library does, but I have no idea :-)
- sergeio76 7y agofefe23, Sorry we did not put all this information into the GitHub readme, we will put more documentation into the doc folder soon, I hope it will answer some of your questions. To specifically answer your questions: Regular expressions are somewhat early POSIX standard... does not have many features that nfa-based regular experssions have like in C#/Python or PCRE … Machines are easy to create but right now it is all done via command line tools, so you will have to write code to create it from code. Does not have JIT. Machines operate on int's (int32), input weight maps and variable length coding is used in places.