4 ms·

What about pictorial languages like Mandarin and Japanese?

• jjtheblunt 8y ago
  Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.
  • rococode 8y ago
    I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch. A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?
  • yomly 8y ago
    Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited? (don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)
• yorwba 8y ago
  Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together. In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.