10 ms·
How does Chrome decide what to highlight when you double-click Japanese text?
- wikibob 6y agoSafari seems to do this too.
- dlivingston 6y agoHere’s a brief write-up [0] on techniques and software for Japanese tokenization. [0]: http://www.solutions.asia/2016/10/japanese-tokenization.html?m=1 http://www.solutions.asia/2016/10/japanese-tokenization.html...
- trnglina 6y agoFirefox, in contrast, breaks at script boundaries, so it'll select runs of Hiragana, Katakana, and Kanji. Not nearly as useful, and definitely makes copying Japanese text especially annoying.
- zeroimpl 6y agoIt also prevents the dictionary lookup gesture of macOS from working in Firefox, since it selects the whole sentence and looks that up (which fails).
- Wowfunhappy 6y agoSide note, this already doesn't work if you don't have a Touchpad / Magic mouse. The normal workflow is Right Click ==> Look Up, but Firefox overrides macOS's normal right click menu. :( Firfox is just not a very good macOS citizen, sadly.
- spacehunt 6y agoYou can still select the text and choose Look Up from the Services menu.
- Wowfunhappy 6y agoOh, there is that, but it's too far away to be useful. If I could set up a custom keyboard shortcut for the service, it would work great—but support for custom keyboard shortcuts also doesn't work in macOS Firefox[1]! (Unless you set a combination which Firefox doesn't already use for something, and it already uses everything sensible, so you have to set something long and annoying.) [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1333781 https://bugzilla.mozilla.org/show_bug.cgi?id=1333781
- knolax 6y agoPersonally I find double click highlighting to be a useless feature in any language, but the Firefox approach is superior imo. Breaking at script boundaries is predictable behavior the user can anticipate whereas doing some janky ad hoc natural language processing invariably results in behavior that is essentially random from a user perspective. I tried out double click highlighting on some basic Japanese sentences on Chromium and it failed to highlight any of what would be considered words. It's not like English highlighting does complex grammatical analysis to make sure "Project Manager" gets highlighted as one chunk and "eventUpdate" gets highlighted as two chunks, most implementations just breaks at spaces like the user expects.
- microcolonel 6y agoI feel like this is a conclusion you could only reach by having an irrational compulsion to defend the deficiencies of Firefox, and not being a regular user of the Japanese language.
- knolax 6y agoI don't even use Firefox. You know what's really irrational? Trying to find ulterior motive in an argument instead of addressing its substance.
- Wowfunhappy 6y agoI don't use/speak Japanese, but the GP's conclusion makes complete sense to me. Consistent behavior > inconsistent behavior, almost always. If I can anticipate what my computer is going to do, I can plan around it, even if it means a bit of extra manual work.
- BigJono 6y agoThat doesn't work if the consistent behaviour is completely useless. If double click selected English tokens grouped by which half of the alphabet they came from, nobody would ever double click anything, they'd just click-drag highlight.
- jack1243star 6y agoI actually like this behavior more, since it is predictable. Sometimes it just works for occasional looking up proper nouns, and you can already tell if it won't.
- trnglina 6y agoI think it depends on how engrained the double-click highlight is for you. For me, I double-click by default, since I almost always want to select at a word boundary in English. As a result, when I need to select Chinese or Japanese text, I'm always annoyed when my double click (which, in my mind, should always select a word) selects a nonsensical sub-sentence instead, and I have to then re-select it manually.
- naniwaduni 6y agoWhen you double-click, say, the word "double-click", do you expect to highlight either "double" or "click", depending on which part of the word I happened to click on? Or do you expect to highlight the entire compound? Does your expectation change when the phrase is written with spacing: "double click"? When written without spacing: "doubleclick"? Does your perception of the word boundaries change between these cases? Now consider another anglophone arbitrarily; how likely does it seem to you that they would share the exact same set of expectations of the above with you? Because if they don't, then one of you is going to have their expectations violated. Now give it a try: do the highlights match your expectations? (Moreover: are those expectations consistent with where you placed the word boundaries?) If your expectations of what double-click highlight does for English is consistent with what actually happens when you double-click, it is deeply unlikely that the causality goes word boundary -> highlight expectation; if they do, it's more likely that double-click highlight has influenced your sense of word boundaries. But it's even more likely that you your expectations of double-click highlight don't actually match up with your intuition for word boundaries, and you just have a pretty good model of how double-click highlight behaves independently of where you think the word boundaries are. (The notion of word boundaries in this context is fraught anyway. Even setting aside the problem of compound words, the phonological word boundaries of Japanese don't match up with the lexicographical ones, in both directions.)
- darkerside 6y ago> The quality is not amazing but I'm surprised this is supported at all. I find this line hilarious for some reason. Reminds me of the line about being a tourist in France, "French people don't expect you to speak French, but they appreciate it when you try"
- whoisjuan 6y agoWas OP referring to the fact that V8 namespace is available inside JSFiddle? Because I was a bit surprised about that and made me wonder if opening this JSFiddle on Safari would work at all (I’m on a phone so I can’t test).
- darkerside 6y agoThat's funny. Not at all the way I read it, but it could totally be read that way. Made me do a double take. Still, I think my original reading is correct, because I don't think there is any issue with the "quality" of v8 inside of jsfiddle. While imagining Chrome doing its best to identify real words in long strings of Japanese text and failing spectacularly just made me laugh again.
- yftsui 6y agoThe fiddle doesn't work on Safari. TypeError: Intl.v8BreakIterator is not a function. (In 'Intl.v8BreakIterator(['ja-JP'], {type:'word'})', 'Intl.v8BreakIterator' is undefined)
- polm23 6y agoOP here, I was not referring to it working JSFiddle, just the idea of Chrome shipping with a Japanese tokenizer. I've known people to do stuff like compile MeCab with emscripten to get Japanese tokenization in Electron, so the fact that the functionality was there already (even with lower quality) was surprising.
- curiousgal 6y agoBeing a resident of France I have to say that French people are the opposite of that. Same for Japanese people. The only people who get genuinely excited as you butcher their language are Arabic speakers I've noticed.
- emilfihlman 6y agoFrom a very quick cursory look, Japanese seems to have some sort of commas and periods. Shouldn't it be just simple to adopt spaces, (and proper commas and periods and parentheses, which they actually use already) too?
- corey_moncure 6y agoSort of but then no one can agree on things like whether particles are part of the word or not. To the fluent reader, the kanji/kana switching breaks up the text enough to lex it. I'm non-native and actually reading all-kana is much more difficult for me than the usual mixture. Many all-kana texts employ spaces. All-kana with no spaces is a nightmare.
- LikeAnElephant 6y agoAgree with this, then you come across a more philosophical debate: should the very nature of written language change to adapt to the needs of technology, or the other way around? On one hand, language has always been an amorphous thing and has changed throughout history. On the other could some changes change the very nature and heart of it? No one agrees on those trade-offs, and I argue likely never will.
- Asooka 6y ago> should the very nature of written language change to adapt to the needs of technology This has been always the case. I mean, think of how modern latin letter forms developed. Think back to the Roman Empire, when the letters were etched in stone or wax tablets, with only capitals (before the invention of minuscule), drawn with straight lines that are easy to chisel. Think of the later periods when many books were written by scribes using pen and ink, resulting in changes to letters so they could write entire words in one hand motion. Later still, we have the printing press and its demands that each letter be a discrete unit, doing away with the wavy flow of words. Today most people write using print letters simply because the majority of writing you would encounter is in print, so that's what's easiest to read. Those changes probably don't register to you as egregious as the one parent proposes, due to the fact that the person enforcing the change and the person developing the technology are one and the same - a native speaker modifying his own language to fit the tools at hand. Which I think brings us to a resolution - the native speakers of the language are the ones to decide how to change it to suit their everyday needs, including making it easier to produce and consume with the tools of the day.
- JonathonW 6y agoICU (International Components for Unicode) provides an API for this: http://userguide.icu-project.org/boundaryanalysis http://userguide.icu-project.org/boundaryanalysis Assuming Blink is using the same technique for text selection as V8 is for the public Intl.v8BreakIterator method, that's how Chrome's handling this-- Intl.v8BreakIterator is a pretty thin wrapper around the ICU BreakIterator implementation: https://chromium.googlesource.com/v8/v8/+/refs/heads/master/src/objects/js-break-iterator.cc#75 https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...
- wwarner 6y agowas looking for that last link myself -- thanks!
- erjiang 6y agoAccording to your first link, BreakIterator uses a dictionary for several languages, including Japanese. So I guess the full answer is something like: Chrome uses v8's Intl.v8BreakIterator which uses icu::BreakIterator, which, for Japanese text, uses a big long list of Japanese words to try to figure out what is a word and what isn't. I've worked on a similar segmenter for Chinese and yeah, quality isn't great but it works in enough cases to be useful.
- peterburkimsher 6y agoWhat did you use for your Chinese word segmentation? I wrote Pingtype myself to help me learn by splitting words and translating them in parallel. https://pingtype.github.io https://pingtype.github.io
- TheChaplain 6y agoFor firefox there's a bug discussing the pros and cons of using ICU for boundaries, so at least they are aware of the issue. https://bugzilla.mozilla.org/show_bug.cgi?id=820261 https://bugzilla.mozilla.org/show_bug.cgi?id=820261
- chch 6y agoDoing a bit more deep diving into the ICU code, it looks like the source code for the Break engine (used by Chinese, Japanese, and Korean) is here: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... and then according to the LICENSE file[1], the dictionary : # The word list in cjdict.txt are generated by combining three word lists # listed below with further processing for compound word breaking. The # frequency is generated with an iterative training against Google web # corpora. # # * Libtabe (Chinese) # - https://sourceforge.net/project/?group_id=1519 # - Its license terms and conditions are shown below. # # * IPADIC (Japanese) # - http://chasen.aist-nara.ac.jp/chasen/distribution.html # - Its license terms and conditions are shown below. # It's interesting to see some of the other techniques used in that engine, such as a special function to figure out the weights of potential katakana word splits. [1] https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7c875520fc8397fa01099/icu4c/LICENSE https://github.com/unicode-org/icu/blob/6417a3b720d8ae3643f7...
- LikeAnElephant 6y agoThis is often determine by Unicode and not the browsers specifically (though some browsers could override the suggested Unicode approach). Each unicode character has certain properties, one of which is whether that character indicates a break before / after itself. I've done extensive research on this for my job, but unfortunately don't have time to do the whole writeup here. Here are several resources for those who are interested Info on break opportunities: https://unicode.org/reports/tr14/#BreakOpportunities https://unicode.org/reports/tr14/#BreakOpportunities The entire Unicode Character Database (~80MB XML file last I checked) https://unicode.org/reports/tr44/ https://unicode.org/reports/tr44/ The properties within the UCD are hard to parse, here's a reference if you're interested: https://unicode.org/reports/tr14/#Table1 https://unicode.org/reports/tr14/#Table1 https://www.unicode.org/Public/5.2.0/ucd/PropertyAliases.txt https://www.unicode.org/Public/5.2.0/ucd/PropertyAliases.txt https://www.unicode.org/Public/5.2.0/ucd/PropertyValueAliases.txt https://www.unicode.org/Public/5.2.0/ucd/PropertyValueAliase... Overall, word / line breaking in Unicode in no-space languages is a very difficult problem. Where the UCD says there can be a line break isn't where a native speaker would put one. In order to do it correctly you have to bring in Natural Language Processing, but that has its own set of complexities. In summary: I18N is hard!
- swang 6y agoYes. This seems to work even when you pass it Chinese while maintaining ja-JP as the language function tokenizeJA(text) { var it = Intl.v8BreakIterator(['ja-JP'], {type:'word'}) it.adoptText(text) var words = [] var cur = 0, prev = 0 while (cur < text.length) { prev = cur cur = it.next() words.push(text.substring(prev, cur)) } return words } console.log(tokenizeJA("今天要去哪裡?")) still seems to parse just fine. so most likely just using the passed input to parse.
- LikeAnElephant 6y agoYep, the browser has the UCD info built into it (a simplification... but basically). Similarly our mobile devices and various backend languages have the same data backed into it. This is where there are sometimes discrepancies between how a given browser or device would output this data, as it could be working off of an outdated version of Unicode's data. Some devices even overwrite the default Unicode behavior. There are just SO many languages and SO many regions and SO many combinations thereof that even Unicode can't cover all the bases. It's all very fascinating from an engineering perspective.
- 1024core 6y agoSo what is the most accurate way to tokenize CJK text?
- edflsafoiewq 6y agoBy hand using a native speaker.
- tobyhinloopen 6y agoIs there an API for that?
- 1024core 6y agoSurely someone must've come up with rules for it by now?
- earthboundkid 6y agoAre there rules (as opposed to heuristics) for splitting up English, like “words—separated haphazardly—by em–dashes?”
- peter303 6y agoI wonder if this applicable to Chinese. In Chinese one to four characters comprise a word. In post-revolutionary Chinese all the characters in a sentence are run together and you have to mentally parse the words in your mind. (Its worse in pre-revolutionary Chinese where neither sentences nor paragraphs are punctuated.) (Pre-medieval european languages used to run all their letters together without word or sentence breaks. Torah Hebrew still does this.)
- peterburkimsher 6y agoYes, I wrote https://pingtype.github.io https://pingtype.github.io for Chinese word segmentation to help me learn.
- knolax 6y agoDouble-click highlighting barely makes any sense in English, let alone in a language that doesn't use spaces. The fact that mobile browsers treat long presses as a double press for urls has been the bane of my existence. I doubt any native Japanese speakers use this feature.
- oh_sigh 6y agoGiven this property of Japanese text, is there wordplay associated with a string of characters with double/reverse meanings depending on how the characters are combined?
- needle0 6y agoYes. One that comes to mind is "[name]さんじゅうななさい" which can be either interpreted as "[name]-san, 17 years old" or "[name], 37 years old", depending on whether you interpret the さん(san) as an honorific or part of a number. (The sentence would usually be written in a combination of hiragana and kanji, but is intentionally written in all hiragana here to ensure ambiguity.)
- needle0 6y agoAnother one: "この先生きのこるには", which should be broken up at "この先/生きのこるには" to mean "To keep surviving going forward", but since 先生 (teacher) is such a common word, it jumps at your eyes and turns the sentence into "この先生/きのこるには" which means the nonsense "For this teacher to mushroom(verb)". Usually this doesn't happen because the "survive" part is written with more kanji as 生き残る, but here it is written in hiragana to make the きのこ(mushroom) part visible and further mess with lexing. In both cases, some liberties have been taken with notation to intentionally encourage silly mis-readings; It happens much less often in ordinary text.
- polm23 6y agoNot exactly double / reverse meanings, but there are several setences that are traditionally used to test Japanese tokenizers. Most of them are tongue twisters, like this one: すももももももももの内 "Japanese plums (sumomo) and peaches are both kinds of peaches" In speech intonation would make the word boundaries here clear, but in writing it looks odd. This isn't a tongue twister, but tokenizers often fail on it: 外国人参政権は難しい問題だ "Voting rights for foreigners is a complex problem." 外国人 (foreigner) / 参政権 (gov participation rights) is the right tokenization, but a common error is to parse it as 外国 (foreign) / 人参 (carrot) / 政権 (political power).
- dirtydroog 6y agoTIL: Some languages do not have spaces
- SenHeng 6y agoFun fact, Japanese originally didn’t have punctuation. It was a European introduction. https://www.tofugu.com/japanese/japanese-punctuation/ https://www.tofugu.com/japanese/japanese-punctuation/
- simplicio 6y agoI've been doing some work parsing Vietnamese text, which has the opposite problem. Compound words (which is most of the vocabulary) are broken up into their components by spaces, indistinquishable from the boundaries between words.
- enos 6y agoIs that why the name of the country is sometimes spelled with a space, "Viet nam"?
- freddie_mercury 6y agoYes, that's how it is written in Vietnamese. To oversimplify: Vietnamese words are a collection of single syllables that are always separated by a space when writing. "Viet Nam" is also, actually, the "official" English way to write it. (Check how the UN puts it on all their stuff.) However, most Europeans don't do that in their languages, so it usually gets written as Vietnam even by Vietnamese when they're writing European languages.
- AlchemistCamp 6y agoWord segmentation in Japanese isn't that difficult, for the most part. The mixed scripts help quite a bit. The really interesting question is how does Chrome decide what to highlight when you double-click Thai text? It's a non-breaking, (baroquely) phonetic script and the training set is much smaller.
- polm23 6y agoOP here, surprised to see this took off. I actually work with Japanese tokenization a lot - I took over maintenance of the most popular Python Mecab wrapper last year, and I have another Cython-based wrapper that I maintain. Word boundary awareness for Japanese is a pretty uncommon feature in applications, so I was surprised to see the feature had been in Chrome all along, even if it's buried and the quality has issues. Anyway, thanks to everyone who tracked down the icu imlementation and the relevant part of Chrome!