2 ms·

I completely agree with you. Basically for v1 I just took my personal rss feeds and used those. Technology has about 15 sources, Business has about 7, Top has a

I completely agree with you. Basically for v1 I just took my personal rss feeds and used those. Technology has about 15 sources, Business has about 7, Top has about 5. Currently, the summaries are categorized on a feed by feed basis (eg everything from the Verge is technology), but I've been messing around with a Bayes classifier. I just need some training data. Each story is summarized the following way: 1) Get link from feed. 2) If the link doesn't exist, scrape the content and image. 3) Break content into sentences (custom NLP based off of regex). 4) Tokenize sentences into words. 5) Porter-stem all words. 6) Run heavily customized LexRank. 7) Return best sentences, use no more than 15% or 3 sentences, whichever is less. Right now, it's just a firehose of data. There's a lot I can do with it, but I'm exploring where to go: summarize the news? Product reviews? Public domain books? Try and hook up with an rss reader company? Build a chrome extension? I really appreciate the feedback!

PaulHoule 13y ago
I love the user interface. My feeling about the algorithm is that it works really well on some stories and poorly on others. For instance, how can you extract 3 salient points from "10 Tens To Spam The Web With A Top Ten List"? Anyway, a key to advances in practical A.I. is being able to change the problem definition to something that is doable AND serves a need. Competitions like Kaggle and TREC attract smart and hard-working competitors but make a real advance only once every couple of years. You want to beat the odds, rather than summarizing anything that comes down the pipe, you can throw out any articles that don't summarize well. If you could get rid of 50% of the strikeouts it would look much better and if you got rid of 80% it's going to be better than a committee of mechanical turks. Shoot me a line if you want some help making this work.
- samsnelling 13y ago
  Paul, I think we just emailed each other. Again, you are exactly right. Some of my problems I know can be improved using a better scraping algorithm. Another idea would be if the article is not at X length, don't try and summarize it. I look foward to talking to you more about it!

PaulHoule 13y ago

I love the user interface. My feeling about the algorithm is that it works really well on some stories and poorly on others. For instance, how can you extract 3 salient points from "10 Tens To Spam The Web With A Top Ten List"? Anyway, a key to advances in practical A.I. is being able to change the problem definition to something that is doable AND serves a need. Competitions like Kaggle and TREC attract smart and hard-working competitors but make a real advance only once every couple of years. You want to beat the odds, rather than summarizing anything that comes down the pipe, you can throw out any articles that don't summarize well. If you could get rid of 50% of the strikeouts it would look much better and if you got rid of 80% it's going to be better than a committee of mechanical turks. Shoot me a line if you want some help making this work.

samsnelling 13y ago
Paul, I think we just emailed each other. Again, you are exactly right. Some of my problems I know can be improved using a better scraping algorithm. Another idea would be if the article is not at X length, don't try and summarize it. I look foward to talking to you more about it!