4 ms·
If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1
by kanobo 6y ago
If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/src/objects/js-break-iterator.cc#75 https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...
- polm23 6y agoSo that works on similar principles to MeCab but it's more limited and performance is... not very good. I wrote about some of the issues with it recently here. https://github.com/polm/fugashi/issues/23 https://github.com/polm/fugashi/issues/23 (Also, if you learned about Chrome's Japaense tokenization on HN, that was me too!)