5 ms·

A Gentle Introduction to Text Summarization in Machine Learning

325 points by ReDeiPirati 8y ago

schemathings 8y ago
Interesting topic and well written .. I kicked the tires a few times with newspaper3k https://github.com/codelucas/newspaper/ https://github.com/codelucas/newspaper/ it has nlp and summary methods that work pretty well, I think I'll peek under the hood to see how it's being done there. Curious to see if your method is an improvement, if so, hey they're both in python!
heyyyouu 8y ago
This is great stuff. One question -- what about applying the role of the word of the sentence into the weighting, along with the frequency. As in general Subject, Verb and DOs are going to be MUCH more important than, say, adverbs and articles. Is that not done because it's harder to automate vs. just frequency?
f055 8y ago
This article is brilliant, thank you!
nathancahill 8y ago
A large part of the effort in text summarization is in the quality of the stemmer. If you're working with English, you're golden, there are several high-quality stemmers available. However, if you're working in a language that doesn't have a stemmer yet, it's a colossal task to write one.
- rpedela 8y ago
  Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work. For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.
- sansnomme 8y ago
  What about pictorial languages like Mandarin and Japanese?
  - jjtheblunt 8y ago
    Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.
    - rococode 8y ago
      I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch. A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?
    - yomly 8y ago
      Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited? (don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)
  - yorwba 8y ago
    Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together. In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.
i_call_solo 8y ago
Why waste time say lot word when few word do trick?
- m-i-l 8y ago
  "Brevity takes effort": "If I had more time, I would have written a shorter letter."
amelius 8y ago
Am I the only one who hates reading grammatically incorrect pieces of text? For me this technology is still in the "unusable" phase, and urgently needs more work.
- commandlinefan 8y ago
  Yeah, there were a LOT of grammatical mistakes in the linked article: "Text summarization it is the technique" "With the presently explosion" "there is need to develop" I'm not trying to be funny here: I'm genuinely skeptical about somebody who's claiming to have some expertise in machine learning as it applies to textual data but didn't apply a grammar checker (MS Word would catch this stuff).
  - EForEndeavour 8y ago
    What little I'd seen from FloydHub has appeared really crisp and polished... Until this article. The poor grammar is all the more painful given the topic of language processing.
    - rememberlenny 8y ago
      The person who managed their articles just left and went to Cruise.
  - ineedasername 8y ago
    It's a little odd, considering things like the Stanford pos tagger that do a decent job of tense detection.
  - ReDeiPirati 8y ago
    Apologies for the errors in the blog post - we published this one before the final proofread. I have just fixed the post. Thank you for your feedback and we hope you find the content useful.
Koffiepoeder 8y ago
This is very similar to how TLDR-bot on reddit summarizes reddit posts or linked articles: https://smmry.com/about https://smmry.com/about The results can be surprisingly good, even for such a basic algorithm.
ColanR 8y ago
I wish there were also tools like this that did paraphrasing. The article mentions the possibility, using deep learning, but doesn't go into any details; and I haven't seen anything that can summarize below the sentence level anywhere else either.
- heyyyouu 8y ago
  Can I ask what you mean by summarize below the sentence level? Thanks!
  - ColanR 8y ago
    See the subsection titled "Abstraction-based summarization" in the article. Basically, instead of copying important sentences verbatim (and spending all your analysis on choosing those sentences), rewrite entire paragraphs into new and unique sentences. No more plagiarism, because nothing is a direct copy.
    - heyyyouu 8y ago
      That's interesting. One thing about plagiarism is that derivative works are considered plagiarism, even if the direct language is no longer used. https://www.copyright.gov/circs/circ14.pdf https://www.copyright.gov/circs/circ14.pdf A court would have to rule, of course, but I wonder if this would actually get around that. That said, as US caselaw currently stands summaries are completely legal to a certain length -- you can't stop someone from summarizing what exists, it's like sharing a fact -- but you can't create a derivative work. So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism, since summary is covered, but if it's longer, derivative could be an issue. Anyway, thank you -- very interesting!
      - ColanR 8y ago
        > So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism Ya, that's what I was trying to get at when I mentioned summarizing paragraphs into sentences. I think we're on the same page there. :)
nshm 8y ago
Abstractive summarization not yet working (like text generation in chatbots), extractive is ok and better done with RNN/BERT/something else neural network based, that's all you need to know.
swiley 8y ago
A gentle introduction to text summarization using a python NLP library.
seeker_ 8y ago
This is a very basic introduction to the topic. I don't understand why it is getting so much traction here. Am I missing something?
- tracer4201 8y ago
  Might be basic for you, but I know very little about the field. I found it very informative. If you don’t have something constructive, why post at all?

325 points by ReDeiPirati 8y ago

Interesting topic and well written .. I kicked the tires a few times with newspaper3k https://github.com/codelucas/newspaper/ https://github.com/codelucas/newspaper/ it has nlp and summary methods that work pretty well, I think I'll peek under the hood to see how it's being done there. Curious to see if your method is an improvement, if so, hey they're both in python!

heyyyouu 8y ago

This is great stuff. One question -- what about applying the role of the word of the sentence into the weighting, along with the frequency. As in general Subject, Verb and DOs are going to be MUCH more important than, say, adverbs and articles. Is that not done because it's harder to automate vs. just frequency?

f055 8y ago

This article is brilliant, thank you!

nathancahill 8y ago

A large part of the effort in text summarization is in the quality of the stemmer. If you're working with English, you're golden, there are several high-quality stemmers available. However, if you're working in a language that doesn't have a stemmer yet, it's a colossal task to write one.

rpedela 8y ago
Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work. For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.
sansnomme 8y ago
What about pictorial languages like Mandarin and Japanese?
- jjtheblunt 8y ago
  Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.
  - rococode 8y ago
    I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch. A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?
  - yomly 8y ago
    Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited? (don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)
- yorwba 8y ago
  Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together. In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.

i_call_solo 8y ago

Why waste time say lot word when few word do trick?

m-i-l 8y ago
"Brevity takes effort": "If I had more time, I would have written a shorter letter."

amelius 8y ago

Am I the only one who hates reading grammatically incorrect pieces of text? For me this technology is still in the "unusable" phase, and urgently needs more work.

commandlinefan 8y ago
Yeah, there were a LOT of grammatical mistakes in the linked article: "Text summarization it is the technique" "With the presently explosion" "there is need to develop" I'm not trying to be funny here: I'm genuinely skeptical about somebody who's claiming to have some expertise in machine learning as it applies to textual data but didn't apply a grammar checker (MS Word would catch this stuff).
- EForEndeavour 8y ago
  What little I'd seen from FloydHub has appeared really crisp and polished... Until this article. The poor grammar is all the more painful given the topic of language processing.
  - rememberlenny 8y ago
    The person who managed their articles just left and went to Cruise.
- ineedasername 8y ago
  It's a little odd, considering things like the Stanford pos tagger that do a decent job of tense detection.
- ReDeiPirati 8y ago
  Apologies for the errors in the blog post - we published this one before the final proofread. I have just fixed the post. Thank you for your feedback and we hope you find the content useful.

Koffiepoeder 8y ago

This is very similar to how TLDR-bot on reddit summarizes reddit posts or linked articles: https://smmry.com/about https://smmry.com/about The results can be surprisingly good, even for such a basic algorithm.

ColanR 8y ago

I wish there were also tools like this that did paraphrasing. The article mentions the possibility, using deep learning, but doesn't go into any details; and I haven't seen anything that can summarize below the sentence level anywhere else either.

heyyyouu 8y ago
Can I ask what you mean by summarize below the sentence level? Thanks!
- ColanR 8y ago
  See the subsection titled "Abstraction-based summarization" in the article. Basically, instead of copying important sentences verbatim (and spending all your analysis on choosing those sentences), rewrite entire paragraphs into new and unique sentences. No more plagiarism, because nothing is a direct copy.
  - heyyyouu 8y ago
    That's interesting. One thing about plagiarism is that derivative works are considered plagiarism, even if the direct language is no longer used. https://www.copyright.gov/circs/circ14.pdf https://www.copyright.gov/circs/circ14.pdf A court would have to rule, of course, but I wonder if this would actually get around that. That said, as US caselaw currently stands summaries are completely legal to a certain length -- you can't stop someone from summarizing what exists, it's like sharing a fact -- but you can't create a derivative work. So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism, since summary is covered, but if it's longer, derivative could be an issue. Anyway, thank you -- very interesting!
    - ColanR 8y ago
      > So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism Ya, that's what I was trying to get at when I mentioned summarizing paragraphs into sentences. I think we're on the same page there. :)

nshm 8y ago

Abstractive summarization not yet working (like text generation in chatbots), extractive is ok and better done with RNN/BERT/something else neural network based, that's all you need to know.

swiley 8y ago

A gentle introduction to text summarization using a python NLP library.

seeker_ 8y ago

This is a very basic introduction to the topic. I don't understand why it is getting so much traction here. Am I missing something?

tracer4201 8y ago
Might be basic for you, but I know very little about the field. I found it very informative. If you don’t have something constructive, why post at all?