4 ms·

Readability has been ported to Python and Php: http://blog.arc90.com/2009/06/20/readability-now-available-in-three-delicious-flavors/ http://blog.arc90.com/2009

Readability has been ported to Python and Php: http://blog.arc90.com/2009/06/20/readability-now-available-in-three-delicious-flavors/ http://blog.arc90.com/2009/06/20/readability-now-available-i... There is also a C# port here: http://code.google.com/p/nreadability/ http://code.google.com/p/nreadability/

Swizec 16y ago
There is also a ruby port here: https://github.com/iterationlabs/ruby-readability https://github.com/iterationlabs/ruby-readability The beauty of our approach is that we didn't have to port it and we can adapt much quicker when new versions come out.
- lamnk 16y ago
  Great, i searched for a readability's ruby port on github but the closest i can find is pismo: https://github.com/peterc/pismo https://github.com/peterc/pismo
  - petercooper 16y ago
    Pismo used to use ruby-readability (linked above) but I ended up writing my own system. It works similarly to Readability but is better on certain types of poorly formatted content (but worse on others, so YMMV). Pismo is more a general purpose content extraction library than Readability and better suited for machine processing and summarization (which is what I use it for). Pismo also comes with a command line client built-in, so you can do stuff like this: $ pismo http://preona.net/2010/11/ever-wanted-arc90s-readability-as-an-api/ title sentences :title: "Ever wanted arc90's Readability as an API?" :sentences: Over at Preona we have been wanting something just like that for a while now. So we built it! Some time ago, while developing LazyReadr, we were faced with the fact that RSS feeds simply aren't all that lovely anymore. Note: "sentences" picks the first few sentences by default, but this is ideal for a summary by an automated system or for a news page :-)
    - Swizec 16y ago
      You should check out topicmarks. It does summaries in a very smart way from what I've seen. We'll likely start using it to do automated summaries for RSS content in our iPad app.
      - petercooper 16y ago
        Sadly, though, it "takes minutes" (and they even seem to make a big point of that..) It might be useful for slightly better summaries though I've had great luck with going with the first paragraph of an article so far (or certain other metadata if it scores better, like <meta> description).
- tectonic 16y ago
  Ruby-readability is one of my projects. Let me know if you have any questions.
kwellman 16y ago
I recently converted the Python port to use lxml instead of BeautifulSoup. If you need speed (BeautifulSoup is slow), and not portability (lxml won't work on google app engine), check out http://blog.interstellr.com/readability-in-python-using-lxml http://blog.interstellr.com/readability-in-python-using-lxml
izendejas 16y ago
I ported it to Java a while back and was amazed at how a simple heuristic (looking for blocks of text with lots of commas) went a long way. Unfortunately, it has several drawbacks which have prevented my full use of it and came up with an algorithm which turned out to be similar to boilerpipe's (using decision trees) that worked better in some cases. I haven't gone back to this problem, but I have some ideas on ways that it can be improved. I actually suggested that there be a <usercontent> tag to one of the people on the HTML5 committee because among other things comments sections throw off many of these web scraping algorithms. After 3 emails, he didn't buy my argument, so I gave up. So all web designers, please oh, pretty please, use the up and coming <article> tag correctly. Many data/text miners will thank you dearly!

Swizec 16y ago

There is also a ruby port here: https://github.com/iterationlabs/ruby-readability https://github.com/iterationlabs/ruby-readability The beauty of our approach is that we didn't have to port it and we can adapt much quicker when new versions come out.

lamnk 16y ago
Great, i searched for a readability's ruby port on github but the closest i can find is pismo: https://github.com/peterc/pismo https://github.com/peterc/pismo
- petercooper 16y ago
  Pismo used to use ruby-readability (linked above) but I ended up writing my own system. It works similarly to Readability but is better on certain types of poorly formatted content (but worse on others, so YMMV). Pismo is more a general purpose content extraction library than Readability and better suited for machine processing and summarization (which is what I use it for). Pismo also comes with a command line client built-in, so you can do stuff like this: $ pismo http://preona.net/2010/11/ever-wanted-arc90s-readability-as-an-api/ title sentences :title: "Ever wanted arc90's Readability as an API?" :sentences: Over at Preona we have been wanting something just like that for a while now. So we built it! Some time ago, while developing LazyReadr, we were faced with the fact that RSS feeds simply aren't all that lovely anymore. Note: "sentences" picks the first few sentences by default, but this is ideal for a summary by an automated system or for a news page :-)
  - Swizec 16y ago
    You should check out topicmarks. It does summaries in a very smart way from what I've seen. We'll likely start using it to do automated summaries for RSS content in our iPad app.
    - petercooper 16y ago
      Sadly, though, it "takes minutes" (and they even seem to make a big point of that..) It might be useful for slightly better summaries though I've had great luck with going with the first paragraph of an article so far (or certain other metadata if it scores better, like <meta> description).
tectonic 16y ago
Ruby-readability is one of my projects. Let me know if you have any questions.

kwellman 16y ago

I recently converted the Python port to use lxml instead of BeautifulSoup. If you need speed (BeautifulSoup is slow), and not portability (lxml won't work on google app engine), check out http://blog.interstellr.com/readability-in-python-using-lxml http://blog.interstellr.com/readability-in-python-using-lxml

izendejas 16y ago

I ported it to Java a while back and was amazed at how a simple heuristic (looking for blocks of text with lots of commas) went a long way. Unfortunately, it has several drawbacks which have prevented my full use of it and came up with an algorithm which turned out to be similar to boilerpipe's (using decision trees) that worked better in some cases. I haven't gone back to this problem, but I have some ideas on ways that it can be improved. I actually suggested that there be a <usercontent> tag to one of the people on the HTML5 committee because among other things comments sections throw off many of these web scraping algorithms. After 3 emails, he didn't buy my argument, so I gave up. So all web designers, please oh, pretty please, use the up and coming <article> tag correctly. Many data/text miners will thank you dearly!