3 ms·

A large part of the effort in text summarization is in the quality of the stemmer. If you're working with English, you're golden, there are several high-quality

A large part of the effort in text summarization is in the quality of the stemmer. If you're working with English, you're golden, there are several high-quality stemmers available. However, if you're working in a language that doesn't have a stemmer yet, it's a colossal task to write one.

rpedela 8y ago
Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work. For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.
sansnomme 8y ago
What about pictorial languages like Mandarin and Japanese?
- jjtheblunt 8y ago
  Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.
  - rococode 8y ago
    I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch. A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?
  - yomly 8y ago
    Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited? (don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)
- yorwba 8y ago
  Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together. In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.

rpedela 8y ago

Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work. For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.

sansnomme 8y ago

What about pictorial languages like Mandarin and Japanese?

jjtheblunt 8y ago
Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.
- rococode 8y ago
  I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch. A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?
- yomly 8y ago
  Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited? (don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)
yorwba 8y ago
Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together. In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.