4 ms·

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

yesimahuman 16y ago
Boilerpipe is by far the best tool for this that I've ever found (http://code.google.com/p/boilerpipe/ http://code.google.com/p/boilerpipe/). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.
- benmccann 16y ago
  I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.
  - yesimahuman 16y ago
    Yea, I'd be curious to see exactly what he's doing. I can only guess there is a heuristic which results in a lot of failed feed processing noticed on here (I know it's just a weekend project :)) that doesn't generalize well. Boilerpipe, in my experience, works very well on almost all news/blog type content. Finding the date in the first few sentences and the title are extra heuristics that can be added later. EDIT: The date and title are in the RSS feed already! No further analysis needed.
- nikcub 16y ago
  Thanks for that link - exactly what I was looking for btw I know that at Techmeme, Gabe spent years perfecting his story parsing for the 50k+ sites he tracks. Even something that would seem simple such as parsing the date of a story from a webpage has a ridiculous number of permutations that you have to grep for.
- gregorym 16y ago
  That is freaking awesome! http://boilerpipe-web.appspot.com/ http://boilerpipe-web.appspot.com/
- osmandros 16y ago
  That library is very robust. On the company I was working last year, We built a news crawler using that tool, and adapted it as a plugin for nutch.
beagledude 16y ago
Goose article extractor has a full suite of unit tests and also does pure text and image extractions: https://github.com/jiminoc/goose https://github.com/jiminoc/goose
k1m 16y ago
Possible to do it with Readability - my PHP port is here: http://www.keyvan.net/2010/08/php-readability/ http://www.keyvan.net/2010/08/php-readability/ - and similar tool (free software, can be self-hosted) using the PHP Readability here: http://fivefilters.org/content-only/ http://fivefilters.org/content-only/

yesimahuman 16y ago

Boilerpipe is by far the best tool for this that I've ever found (http://code.google.com/p/boilerpipe/ http://code.google.com/p/boilerpipe/). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

benmccann 16y ago
I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.
- yesimahuman 16y ago
  Yea, I'd be curious to see exactly what he's doing. I can only guess there is a heuristic which results in a lot of failed feed processing noticed on here (I know it's just a weekend project :)) that doesn't generalize well. Boilerpipe, in my experience, works very well on almost all news/blog type content. Finding the date in the first few sentences and the title are extra heuristics that can be added later. EDIT: The date and title are in the RSS feed already! No further analysis needed.
nikcub 16y ago
Thanks for that link - exactly what I was looking for btw I know that at Techmeme, Gabe spent years perfecting his story parsing for the 50k+ sites he tracks. Even something that would seem simple such as parsing the date of a story from a webpage has a ridiculous number of permutations that you have to grep for.
gregorym 16y ago
That is freaking awesome! http://boilerpipe-web.appspot.com/ http://boilerpipe-web.appspot.com/
osmandros 16y ago
That library is very robust. On the company I was working last year, We built a news crawler using that tool, and adapted it as a plugin for nutch.

beagledude 16y ago

Goose article extractor has a full suite of unit tests and also does pure text and image extractions: https://github.com/jiminoc/goose https://github.com/jiminoc/goose

k1m 16y ago

Possible to do it with Readability - my PHP port is here: http://www.keyvan.net/2010/08/php-readability/ http://www.keyvan.net/2010/08/php-readability/ - and similar tool (free software, can be self-hosted) using the PHP Readability here: http://fivefilters.org/content-only/ http://fivefilters.org/content-only/