4 ms·
Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in. Eg https:/
by epcoa 2y ago
Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in.
Eg https://github.com/wangfenjin/simple https://github.com/wangfenjin/simple
- wahnfrieden 2y agoI know it’s extensible, with original work. ICU isn’t helpful for some languages
- AlexeyBelov 2y agoFor what languages? Why ICU isn't helpful? My naive guess would be "it works for anything supported by Unicode".
- wahnfrieden 2y agolanguages that don't use spaces like Japanese