5 ms·

Cool! So, would it be correct to say that this is in essence generates a disambiguation model for a language with a lot of homonyms due to having relatively few

Cool! So, would it be correct to say that this is in essence generates a disambiguation model for a language with a lot of homonyms due to having relatively few sounds (but also with some variation in morpheme boundaries, e.g. "an-i" vs "ani")?

microcolonel 9y ago
For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel. P.S. Mandarin has a fair few homophones as well (yes, tones and all). English has tonnes of phonemes, and still we have loads of homophones (and in some of the most common words, no less!). Japanese, to my elementary-level ear, doesn't sound an order of magnitude more ambiguous than English.
- blipmusic 9y ago
  Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discovered (apparently well-known - something about a horse and a...mother?). If tones are not represented in romanised Chinese, things seem to get tricky, indeed.
  - microcolonel 9y ago
    For what it's worth, there's a lot of work to do on Japanese input methods. To a human it may be obvious that hawotogu should become 刃を研ぐ (sharpen a/the blade), and that one does not typically sharpen or hone a 歯 (tooth), 葉(leaf), 派(party/ingroup), or even 覇(~hegemony); but to a computer without topic-specific context, each of these is equally valid. One might be thankful that humans don't really say all that many different or interesting things, at least in this context.
    - derefr 9y ago
      I don't know about "topic-specific context", but doesn't a word-granular Hidden Markov Model trained on huge amounts of text from the language let you figure out things like "given that the sentence includes 刃, 研 is the highest-scoring match"?
      - microcolonel 9y ago
        It does, but the suggestions will be nowhere to be found if somebody is making a novel use of words. That's what I was getting at.
    - blipmusic 9y ago
      I didn't mean to imply otherwise. My curiosity was/is with the generation of the model in the link (especially compared to current models), hence the question in my original post. There's also pitch accent that helps with disambiguation of spoken Japanese (I should know, I accidentally called a friend seaweed for a long time).
  - schoen 9y ago
    I don't speak Chinese but I've encountered this series a lot: https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese 媽 mā 'mother' https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese 麻 má 'hemp' (sometimes 'flax') https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese 馬 mǎ 'horse' https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese 罵 mà 'scold' It's also cool that you can see that 媽 is made up of "semantic 女 + phonetic 馬", where 女 means 'lady' and 馬 sounds like "ma", so the character was meant to suggest "a word relating to ladies that sounds like ma". https://en.wikipedia.org/wiki/Chinese_character_classification#Phono-semantic_compound_characters https://en.wikipedia.org/wiki/Chinese_character_classificati...
    - PeCaN 9y ago
      There's also this rather fun poem, Shī Shì shí shī shǐ. https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_Den https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...
    - brobinson 9y ago
      There's one more: https://en.wiktionary.org/wiki/%E5%97%8E#Chinese https://en.wiktionary.org/wiki/%E5%97%8E#Chinese 嗎 ma 'question particle'
      - schoen 9y ago
        Oh yeah, thanks! Apparently the phono-semantic derivation for that is "mouth ma" (maybe because a mouth is used to ask questions?).
  - brobinson 9y ago
    Was the phrase possibly this: 你敢罵我媽的馬嗎？ (roughly: "you dare to scold my mother's horse?")
    - blipmusic 9y ago
      "Scold" or something similar to it was in there, so quite possibly yes. Thanks, for the heads up.
- derefr 9y ago
  Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and required people to hit the spacebar between the "words" (lexer tokens) of the provided input for matching, they'd have a much simpler job. But as it is, they fall over quite badly when you type multiple words into the IME input box, and are mostly only usable if you resolve single words at a time (which, sadly, throws away a lot of the inter-word context that would otherwise be available for matching.)
  - gizmo686 9y ago
    >because Japanese people pause between spoken words just like anyone else. This is (surprisingly) not true. People do not pause between words, however when listening to a language that they understand, they do perceive pauses between words; even though such pauses do not exist.
    - yomly 9y ago
      I disagree. There's a subtle difference between breathing cadence and inflection, and a completely monotempo monotonal string of sounds.
      - snakeboy 9y ago
        It seems like it'd be useful for one or both of you to cite any research than has been done on this. Seems more productive (and enlightening to all) than the agree/disagree dialogue here.
        gizmo686 9y ago
        Unfourtuantly, this is so well established that it is hard to find research. I did find this paper [1] which looks into how babies acuire word boundries. As you identify, there are probably phonetic cues; but not pauses. The best way to see this is to try listening to a language you do not understand, and try to identify word boundries. Indeed, the paper I link argues that some phonetic cue must exist because babies can recognise word boundries. [1] https://www.ncbi.nlm.nih.gov/pubmed/8176060 https://www.ncbi.nlm.nih.gov/pubmed/8176060
- fenomas 9y ago
  > Japanese, to my elementary-level ear, doesn't sound an order of magnitude more ambiguous than English I don't have any citations, but subjectively I disagree. One can think of English words that are homophones, but in JP the challenge is more to think of words that aren't. On one hand, if you include uncommon words (as a keyboard's corpus would) then practically any medium-length word will have multiple possibilities, which is not the case in English. But much more important is that, where an English homophone typically has two or possibly three interpretations, a kanji jukugo might have 3-4 everyday meanings and a bunch more uncommon ones. And all this is on top of the matter of word boundaries. If the user enters し, there could be 10+ possible transliterations of that character as a standalone word/particle, above and beyond whatever readings are possible together with the characters before and after. So I don't know what an order of magnitude would mean in this context, but I think the whole matter is significantly more ambiguous than English.

microcolonel 9y ago

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel. P.S. Mandarin has a fair few homophones as well (yes, tones and all). English has tonnes of phonemes, and still we have loads of homophones (and in some of the most common words, no less!). Japanese, to my elementary-level ear, doesn't sound an order of magnitude more ambiguous than English.

blipmusic 9y ago
Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discovered (apparently well-known - something about a horse and a...mother?). If tones are not represented in romanised Chinese, things seem to get tricky, indeed.
- microcolonel 9y ago
  For what it's worth, there's a lot of work to do on Japanese input methods. To a human it may be obvious that hawotogu should become 刃を研ぐ (sharpen a/the blade), and that one does not typically sharpen or hone a 歯 (tooth), 葉(leaf), 派(party/ingroup), or even 覇(~hegemony); but to a computer without topic-specific context, each of these is equally valid. One might be thankful that humans don't really say all that many different or interesting things, at least in this context.
  - derefr 9y ago
    I don't know about "topic-specific context", but doesn't a word-granular Hidden Markov Model trained on huge amounts of text from the language let you figure out things like "given that the sentence includes 刃, 研 is the highest-scoring match"?
    - microcolonel 9y ago
      It does, but the suggestions will be nowhere to be found if somebody is making a novel use of words. That's what I was getting at.
  - blipmusic 9y ago
    I didn't mean to imply otherwise. My curiosity was/is with the generation of the model in the link (especially compared to current models), hence the question in my original post. There's also pitch accent that helps with disambiguation of spoken Japanese (I should know, I accidentally called a friend seaweed for a long time).
- schoen 9y ago
  I don't speak Chinese but I've encountered this series a lot: https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese 媽 mā 'mother' https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese 麻 má 'hemp' (sometimes 'flax') https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese 馬 mǎ 'horse' https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese 罵 mà 'scold' It's also cool that you can see that 媽 is made up of "semantic 女 + phonetic 馬", where 女 means 'lady' and 馬 sounds like "ma", so the character was meant to suggest "a word relating to ladies that sounds like ma". https://en.wikipedia.org/wiki/Chinese_character_classification#Phono-semantic_compound_characters https://en.wikipedia.org/wiki/Chinese_character_classificati...
  - PeCaN 9y ago
    There's also this rather fun poem, Shī Shì shí shī shǐ. https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_Den https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...
  - brobinson 9y ago
    There's one more: https://en.wiktionary.org/wiki/%E5%97%8E#Chinese https://en.wiktionary.org/wiki/%E5%97%8E#Chinese 嗎 ma 'question particle'
    - schoen 9y ago
      Oh yeah, thanks! Apparently the phono-semantic derivation for that is "mouth ma" (maybe because a mouth is used to ask questions?).
- brobinson 9y ago
  Was the phrase possibly this: 你敢罵我媽的馬嗎？ (roughly: "you dare to scold my mother's horse?")
  - blipmusic 9y ago
    "Scold" or something similar to it was in there, so quite possibly yes. Thanks, for the heads up.
derefr 9y ago
Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and required people to hit the spacebar between the "words" (lexer tokens) of the provided input for matching, they'd have a much simpler job. But as it is, they fall over quite badly when you type multiple words into the IME input box, and are mostly only usable if you resolve single words at a time (which, sadly, throws away a lot of the inter-word context that would otherwise be available for matching.)
- gizmo686 9y ago
  >because Japanese people pause between spoken words just like anyone else. This is (surprisingly) not true. People do not pause between words, however when listening to a language that they understand, they do perceive pauses between words; even though such pauses do not exist.
  - yomly 9y ago
    I disagree. There's a subtle difference between breathing cadence and inflection, and a completely monotempo monotonal string of sounds.
    - snakeboy 9y ago
      It seems like it'd be useful for one or both of you to cite any research than has been done on this. Seems more productive (and enlightening to all) than the agree/disagree dialogue here.
      - gizmo686 9y ago
        Unfourtuantly, this is so well established that it is hard to find research. I did find this paper [1] which looks into how babies acuire word boundries. As you identify, there are probably phonetic cues; but not pauses. The best way to see this is to try listening to a language you do not understand, and try to identify word boundries. Indeed, the paper I link argues that some phonetic cue must exist because babies can recognise word boundries. [1] https://www.ncbi.nlm.nih.gov/pubmed/8176060 https://www.ncbi.nlm.nih.gov/pubmed/8176060
fenomas 9y ago
> Japanese, to my elementary-level ear, doesn't sound an order of magnitude more ambiguous than English I don't have any citations, but subjectively I disagree. One can think of English words that are homophones, but in JP the challenge is more to think of words that aren't. On one hand, if you include uncommon words (as a keyboard's corpus would) then practically any medium-length word will have multiple possibilities, which is not the case in English. But much more important is that, where an English homophone typically has two or possibly three interpretations, a kanji jukugo might have 3-4 everyday meanings and a bunch more uncommon ones. And all this is on top of the matter of word boundaries. If the user enters し, there could be 10+ possible transliterations of that character as a standalone word/particle, above and beyond whatever readings are possible together with the characters before and after. So I don't know what an order of magnitude would mean in this context, but I think the whole matter is significantly more ambiguous than English.