3 ms·

This sure looks interesting. What are the theoretical foundations of this? As for SBS I found this paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.

This sure looks interesting. What are the theoretical foundations of this? As for SBS I found this paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.6530&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222... . I couldn't find anything relevant on DBS, though. At a cursory glance the algorithm seems like a variation Luhn's abstract algorithm.

MojoJolo 13y ago
You are spot on on the paper that I referenced. As you can see in 4.3 which is in page 3, the paper mentioned two algorithms for sentence selection. These are Summation-Based Selection and Density-Based Selection. Which is SBS and DBS respectively.
- agibsonccc 13y ago
  What made you pick this representation in particular? I'm kind of curious what different kinds of algorithms you might have looked at. Summarizing only blogs posts seems a bit limiting to me. (Btw, I'm not trying to be negative, congrats on your success! texteaser looks great!) I implemented a custom version (mainly changed the scoring scheme to include TF/IDF of words for initialized scoring) of TextRank and loved it. The main thing I liked about it was how general it was. Words are nodes and sentences are vertices. Then you basically use pagerank to rank the sentences according the graph representation. [1] http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf
  - MojoJolo 13y ago
    Hi, I focus on blog posts because I don't want it to be broad. This was because TextTeaser is my research for my graduate studies. And having a broader research means harder to accomplish. But it doesn't mean it can't be used to other type of text. It can still be used. It's just optimized for news. I'm a little bit familiar with TextRank because I stumbled upon it when I'm doing my research. I also read several algorithms but forgot what they are called.
    - agibsonccc 13y ago
      Ahh very cool! Thank you for the insight. I could see where that would be applicable then. Using comments as features is a very neat concept. News is the most broadly applicable use for this so leveraging that isn't a bad thing. There's always a trade off of broad applicability vs overfitting for a particular case to get better results. Thanks for the insight! Again great work.
- aswanson 13y ago
  Any suggestions on papers for Luhn's abstract algorithm? I hadn't heard of it before.
  - BjoernKW 13y ago
    There you go: https://text-analysis.googlecode.com/files/luhn58.pdf‎ https://text-analysis.googlecode.com/files/luhn58.pdf‎ http://dl.acm.org/citation.cfm?id=1662360 http://dl.acm.org/citation.cfm?id=1662360
drakaal 13y ago
This was also the foundation for Summly. The problem is that it is flawed. It doesn't take in to account emotion, or emphasis. What is the most important sentence in this: Drakaal is a poopy head. He often posts to hackernews and calls people an idiot. When this happens I get mad. The Premise is "Drakaal is a poopy head" so that is the most important, but it doesn't inform the user. What he does is the most telling of the sentences, but with the "He" as the first word you can't actually make sense of the content with out the prior sentence. The last sentence is the least important for the understanding of the content. It is important to know that sentence number 2 is the most informative, but that to figure out what it means requires Sentence 1. When computing the results of a summary you have to weigh sentence dependencies, density of information, amount of emotion expressed and number of characters available to you. And Keywords aren't enough, you need noun entities and the ability to tell the relationships of words so that you know "Cars, Trucks, and Automobiles" are all the same concept in many contexts.

MojoJolo 13y ago

You are spot on on the paper that I referenced. As you can see in 4.3 which is in page 3, the paper mentioned two algorithms for sentence selection. These are Summation-Based Selection and Density-Based Selection. Which is SBS and DBS respectively.

agibsonccc 13y ago
What made you pick this representation in particular? I'm kind of curious what different kinds of algorithms you might have looked at. Summarizing only blogs posts seems a bit limiting to me. (Btw, I'm not trying to be negative, congrats on your success! texteaser looks great!) I implemented a custom version (mainly changed the scoring scheme to include TF/IDF of words for initialized scoring) of TextRank and loved it. The main thing I liked about it was how general it was. Words are nodes and sentences are vertices. Then you basically use pagerank to rank the sentences according the graph representation. [1] http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf
- MojoJolo 13y ago
  Hi, I focus on blog posts because I don't want it to be broad. This was because TextTeaser is my research for my graduate studies. And having a broader research means harder to accomplish. But it doesn't mean it can't be used to other type of text. It can still be used. It's just optimized for news. I'm a little bit familiar with TextRank because I stumbled upon it when I'm doing my research. I also read several algorithms but forgot what they are called.
  - agibsonccc 13y ago
    Ahh very cool! Thank you for the insight. I could see where that would be applicable then. Using comments as features is a very neat concept. News is the most broadly applicable use for this so leveraging that isn't a bad thing. There's always a trade off of broad applicability vs overfitting for a particular case to get better results. Thanks for the insight! Again great work.
aswanson 13y ago
Any suggestions on papers for Luhn's abstract algorithm? I hadn't heard of it before.
- BjoernKW 13y ago
  There you go: https://text-analysis.googlecode.com/files/luhn58.pdf‎ https://text-analysis.googlecode.com/files/luhn58.pdf‎ http://dl.acm.org/citation.cfm?id=1662360 http://dl.acm.org/citation.cfm?id=1662360

drakaal 13y ago

This was also the foundation for Summly. The problem is that it is flawed. It doesn't take in to account emotion, or emphasis. What is the most important sentence in this: Drakaal is a poopy head. He often posts to hackernews and calls people an idiot. When this happens I get mad. The Premise is "Drakaal is a poopy head" so that is the most important, but it doesn't inform the user. What he does is the most telling of the sentences, but with the "He" as the first word you can't actually make sense of the content with out the prior sentence. The last sentence is the least important for the understanding of the content. It is important to know that sentence number 2 is the most informative, but that to figure out what it means requires Sentence 1. When computing the results of a summary you have to weigh sentence dependencies, density of information, amount of emotion expressed and number of characters available to you. And Keywords aren't enough, you need noun entities and the ability to tell the relationships of words so that you know "Cars, Trucks, and Automobiles" are all the same concept in many contexts.