4 ms·

ICU (International Components for Unicode) provides an API for this: http://userguide.icu-project.org/boundaryanalysis http://userguide.icu-project.org/boundar

ICU (International Components for Unicode) provides an API for this: http://userguide.icu-project.org/boundaryanalysis http://userguide.icu-project.org/boundaryanalysis Assuming Blink is using the same technique for text selection as V8 is for the public Intl.v8BreakIterator method, that's how Chrome's handling this-- Intl.v8BreakIterator is a pretty thin wrapper around the ICU BreakIterator implementation: https://chromium.googlesource.com/v8/v8/+/refs/heads/master/src/objects/js-break-iterator.cc#75 https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...

wwarner 6y ago
was looking for that last link myself -- thanks!
erjiang 6y ago
According to your first link, BreakIterator uses a dictionary for several languages, including Japanese. So I guess the full answer is something like: Chrome uses v8's Intl.v8BreakIterator which uses icu::BreakIterator, which, for Japanese text, uses a big long list of Japanese words to try to figure out what is a word and what isn't. I've worked on a similar segmenter for Chinese and yeah, quality isn't great but it works in enough cases to be useful.
- peterburkimsher 6y ago
  What did you use for your Chinese word segmentation? I wrote Pingtype myself to help me learn by splitting words and translating them in parallel. https://pingtype.github.io https://pingtype.github.io
TheChaplain 6y ago
For firefox there's a bug discussing the pros and cons of using ICU for boundaries, so at least they are aware of the issue. https://bugzilla.mozilla.org/show_bug.cgi?id=820261 https://bugzilla.mozilla.org/show_bug.cgi?id=820261
chch 6y ago
Doing a bit more deep diving into the ICU code, it looks like the source code for the Break engine (used by Chinese, Japanese, and Korean) is here: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... and then according to the LICENSE file[1], the dictionary : # The word list in cjdict.txt are generated by combining three word lists # listed below with further processing for compound word breaking. The # frequency is generated with an iterative training against Google web # corpora. # # * Libtabe (Chinese) # - https://sourceforge.net/project/?group_id=1519 # - Its license terms and conditions are shown below. # # * IPADIC (Japanese) # - http://chasen.aist-nara.ac.jp/chasen/distribution.html # - Its license terms and conditions are shown below. # It's interesting to see some of the other techniques used in that engine, such as a special function to figure out the weights of potential katakana word splits. [1] https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7c875520fc8397fa01099/icu4c/LICENSE https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7...

wwarner 6y ago

was looking for that last link myself -- thanks!

erjiang 6y ago

According to your first link, BreakIterator uses a dictionary for several languages, including Japanese. So I guess the full answer is something like: Chrome uses v8's Intl.v8BreakIterator which uses icu::BreakIterator, which, for Japanese text, uses a big long list of Japanese words to try to figure out what is a word and what isn't. I've worked on a similar segmenter for Chinese and yeah, quality isn't great but it works in enough cases to be useful.

peterburkimsher 6y ago
What did you use for your Chinese word segmentation? I wrote Pingtype myself to help me learn by splitting words and translating them in parallel. https://pingtype.github.io https://pingtype.github.io

TheChaplain 6y ago

For firefox there's a bug discussing the pros and cons of using ICU for boundaries, so at least they are aware of the issue. https://bugzilla.mozilla.org/show_bug.cgi?id=820261 https://bugzilla.mozilla.org/show_bug.cgi?id=820261

chch 6y ago

Doing a bit more deep diving into the ICU code, it looks like the source code for the Break engine (used by Chinese, Japanese, and Korean) is here: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... and then according to the LICENSE file[1], the dictionary : # The word list in cjdict.txt are generated by combining three word lists # listed below with further processing for compound word breaking. The # frequency is generated with an iterative training against Google web # corpora. # # * Libtabe (Chinese) # - https://sourceforge.net/project/?group_id=1519 # - Its license terms and conditions are shown below. # # * IPADIC (Japanese) # - http://chasen.aist-nara.ac.jp/chasen/distribution.html # - Its license terms and conditions are shown below. # It's interesting to see some of the other techniques used in that engine, such as a special function to figure out the weights of potential katakana word splits. [1] https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7c875520fc8397fa01099/icu4c/LICENSE https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7...