3 ms·

I wrote a Chinese Segmenter that is available on the web: https://chinese.yabla.com/chinese-english-pinyin-dictionary.php?define=%E6%88%91%E6%98%AF%E7%94%B5%E8

I wrote a Chinese Segmenter that is available on the web: https://chinese.yabla.com/chinese-english-pinyin-dictionary.php?define=%E6%88%91%E6%98%AF%E7%94%B5%E8%84%91%E7%A8%8B%E5%BA%8F%E5%91%98 https://chinese.yabla.com/chinese-english-pinyin-dictionary.... It does basic path finding, and then picks the best path based on the following rules: 1) Fewest words 2) Least variance in word length (e.g. prefer a 2,2 character split vs a 3-1 split) 3) Solo Freedom (this is based on corpus analysis which tags characters with a probability of being a 1 character word. For example 王家庭 (this is either "Wang Household" (王家庭) or "Prince's courtyard" (王家庭) and we split as Wang Household, because Wang 王 is a common name that frequently appears in isolation, and 庭 is less likely to be in isolation. It is interesting that solo freedom works better than comparing the corpus frequency of "Prince" 王家 vs "Household" 家庭. It works reasonably well. A surprising number of people use it every day.

thaumasiotes 6y ago
> the corpus frequency of "Prince" 王家 What? 王家 doesn't mean "prince". Or at least, there is no such dictionary entry in the ABC dictionary or in the 汉语大词典. I would expect 王家 to mean "prince's household", in the same way that 皇家 means "imperial household". 汉语大词典 has two glosses for 王家: 1. 犹王室，王朝，朝廷。 [Equivalent to "royal family"/"royal court".] 2. 王侯之家。 [An aristocratic household.] There's no problem with the concept of the phrase 王家庭 meaning "prince's courtyard", since a home can easily contain a courtyard. But the phrase should arguably be segmented 王-家-庭. (Or not -- there's very little to distinguish the idea 'one word, "the prince's household"' from 'two words, "prince"/"household"'.) Regardless of that choice, the courtyard is being associated with a household, not a person.
- numpad0 6y ago
  So to illustrate the situation the input would be like “Royal, House, Hold” and whether it’s supposed to be “Royal-Household” or “Mr. Royal’ household” or “Royal House, hold ... ” is up to context right
  - thaumasiotes 6y ago
    Correct, though just to be clear, "hold" doesn't correspond to anything in the Chinese. (I only mention this because on a character-by-character basis, "royal" and "house" are fairly decent glosses of 王 and 家.) "King" is also a decently well-known English surname, so we can draw a pretty close analogy to the distinction between 王家, the royal household, and 王家, the 王 family. Compare the English sentences 1. That's the king's house. [The king lives there] 2. That's the Kings' house. [The Kings live there]
    - jcampbell1 6y ago
      My Chinese ability rounds down to non-existant, and I apologize for constructing a questionable example. Wang as a family name, and Wang as king or (royal-) was the first thing that came to mind. Since you know Chinese well, do you know of a good example of a 1-2, vs 2-1 split where both are reasonable but one is much more likely to be semantically correct? I have been told there are quite long sentences that humorously can be segmented two ways and have completely different meanings, but I can't find any. My favorite in English is expertsexchange.com had to change the domain to experts-exchange.com :)
      - thaumasiotes 6y ago
        > I apologize for constructing a questionable example No need; as an example of segmentation, the one you've presented is fine. The biggest problem, mostly irrelevant, is that "prince's courtyard" is pretty archaic. I was objecting to the translation-in-passing of 王家 as "prince". It's defensible to segment the "prince's courtyard" sense 1-1-1, but it's also defensible to segment it 2-1. I don't have an example of the type of you're looking for to hand, but I will see about finding one.
peterburkimsher 6y ago
Thanks for sharing! I also wrote a Chinese segmentation and parallel translation program, with colours for the tones. Mine only takes one definition for the literal translation though, unlike yours which only handles a single line but with more translations. https://pingtype.github.io https://pingtype.github.io

thaumasiotes 6y ago

> the corpus frequency of "Prince" 王家 What? 王家 doesn't mean "prince". Or at least, there is no such dictionary entry in the ABC dictionary or in the 汉语大词典. I would expect 王家 to mean "prince's household", in the same way that 皇家 means "imperial household". 汉语大词典 has two glosses for 王家: 1. 犹王室，王朝，朝廷。 [Equivalent to "royal family"/"royal court".] 2. 王侯之家。 [An aristocratic household.] There's no problem with the concept of the phrase 王家庭 meaning "prince's courtyard", since a home can easily contain a courtyard. But the phrase should arguably be segmented 王-家-庭. (Or not -- there's very little to distinguish the idea 'one word, "the prince's household"' from 'two words, "prince"/"household"'.) Regardless of that choice, the courtyard is being associated with a household, not a person.

numpad0 6y ago
So to illustrate the situation the input would be like “Royal, House, Hold” and whether it’s supposed to be “Royal-Household” or “Mr. Royal’ household” or “Royal House, hold ... ” is up to context right
- thaumasiotes 6y ago
  Correct, though just to be clear, "hold" doesn't correspond to anything in the Chinese. (I only mention this because on a character-by-character basis, "royal" and "house" are fairly decent glosses of 王 and 家.) "King" is also a decently well-known English surname, so we can draw a pretty close analogy to the distinction between 王家, the royal household, and 王家, the 王 family. Compare the English sentences 1. That's the king's house. [The king lives there] 2. That's the Kings' house. [The Kings live there]
  - jcampbell1 6y ago
    My Chinese ability rounds down to non-existant, and I apologize for constructing a questionable example. Wang as a family name, and Wang as king or (royal-) was the first thing that came to mind. Since you know Chinese well, do you know of a good example of a 1-2, vs 2-1 split where both are reasonable but one is much more likely to be semantically correct? I have been told there are quite long sentences that humorously can be segmented two ways and have completely different meanings, but I can't find any. My favorite in English is expertsexchange.com had to change the domain to experts-exchange.com :)
    - thaumasiotes 6y ago
      > I apologize for constructing a questionable example No need; as an example of segmentation, the one you've presented is fine. The biggest problem, mostly irrelevant, is that "prince's courtyard" is pretty archaic. I was objecting to the translation-in-passing of 王家 as "prince". It's defensible to segment the "prince's courtyard" sense 1-1-1, but it's also defensible to segment it 2-1. I don't have an example of the type of you're looking for to hand, but I will see about finding one.

peterburkimsher 6y ago

Thanks for sharing! I also wrote a Chinese segmentation and parallel translation program, with colours for the tones. Mine only takes one definition for the literal translation though, unlike yours which only handles a single line but with more translations. https://pingtype.github.io https://pingtype.github.io