4 ms·
According to your first link, BreakIterator uses a dictionary for several languages, including Japanese. So I guess the full answer is something like: Chrome u
by erjiang 6y ago
According to your first link, BreakIterator uses a dictionary for several languages, including Japanese. So I guess the full answer is something like:
Chrome uses v8's Intl.v8BreakIterator which uses icu::BreakIterator, which, for Japanese text, uses a big long list of Japanese words to try to figure out what is a word and what isn't. I've worked on a similar segmenter for Chinese and yeah, quality isn't great but it works in enough cases to be useful.
- peterburkimsher 6y agoWhat did you use for your Chinese word segmentation? I wrote Pingtype myself to help me learn by splitting words and translating them in parallel. https://pingtype.github.io https://pingtype.github.io