3 ms·
Excellent post on the basics! As you mentioned in your post, the real work begins once you have the algorithm in place. Scraping is a challenge as Readability,
by samsnelling 13y ago
Excellent post on the basics! As you mentioned in your post, the real work begins once you have the algorithm in place.
Scraping is a challenge as Readability, boiler pipe and so on are only so effective.
Sentence parsing is extremely tricky. Support for surnames, urls, ellipsis, quotations, and all the other weird language things.
This is the simplest implementation of textrank I've seen, and a lot of credit to the author for writing it in such a way. If anyone is interested, you can find out more about textrank from this academic paper which explains it quite nicely (http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf).
Note: just released http://summary.io http://summary.io
- MojoJolo 13y agoHi, I just saw your http://summary.io http://summary.io. I also have a challenge in extracting the text from the website. I used boilerpipe, but it doens't perform well for me. I then changed it to Diffbot (http://diffbot.com/ http://diffbot.com/). It extracts the text way better but there is still some minor problems. May I know what you are using? Additional question is how you extract the images in the text? Btw, your http://summary.io http://summary.io is almost the same as my http://readborg.com/ http://readborg.com/ (sorry, login required and it's still invitation only).
- shlomib 13y agoAs I wrote in the post, I use Goose (https://github.com/xgdlm/python-goose https://github.com/xgdlm/python-goose) in my product. Goose also returns the URL of the main image! (sometimes it even includes its dimensions)