8 ms·

Any developers who'd like to contribute to improving how article content is extracted from web pages should check out Mozilla's Readability repository: https://

by k1m 5y ago

Any developers who'd like to contribute to improving how article content is extracted from web pages should check out Mozilla's Readability repository: https://github.com/mozilla/readability https://github.com/mozilla/readability I'm currently trying to bring the PHP port up to speed here: https://github.com/fivefilters/readability.php https://github.com/fivefilters/readability.php We use an older version as part of our article extraction for Push to Kindle: https://www.fivefilters.org/push-to-kindle/ https://www.fivefilters.org/push-to-kindle/

infogulch 5y ago
ArchiveBox is a tool that downloads web pages and saves them in various different formats: warc, pdf, rendered png, plain text. I wonder what it uses for plain text extraction and if the readability repo would be useful for that purpose. Edit: Oh neat it does actually. https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_readability https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#... > Archive method SAVE_READABILITY > Extract article text, summary, and byline using Mozilla's Readability library. Unlike the other methods, this does not download any additional files, so it's practically free from a disk usage perspective. It works by using any existing downloaded HTML version (e.g. wget, DOM dump, singlefile) and piping it into readability.
- jrochkind1 5y ago
  That's pretty amazing it already does it. ArchiveBox and the other stuff from the "DIY no-credentials don't-care-about-the-rules" web archiving community, like ArchiveTeam.... continues to astound me with it's quality and "professionalism" (as a credentialed professional in the field of digital library stuff... they are often outdoing the actual credentialed professional community).
  - nikisweeting 5y ago
    Aww thank you!
up6w6 5y ago
I believe the Instant View[1] crowdsourcing model where people write templates for each website could boost a lot these parsers (hope they open source it soon). Its just impossible to make these extensions work for every single website with some simple heuristics. Check the codebase of some popular parsers: Firefox (already mentioned): https://github.com/mozilla/readability/blob/master/Readability.js https://github.com/mozilla/readability/blob/master/Readabili... Google Chrome: https://github.com/chromium/dom-distiller https://github.com/chromium/dom-distiller Mercury parser: https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser [1] https://instantview.telegram.org/ https://instantview.telegram.org/
- k1m 5y ago
  Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.
  - freediver 5y ago
    Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?
    - k1m 5y ago
      In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
      - freediver 5y ago
jamil7 5y ago
I wrote a Swift port of it last year for my app but it deviates from readability a fair bit as I tailed it a bit to my needs, I've considered cleaning it up and open sourcing it regardless. I know there is an Objective-C port floating around.
- freediver 5y ago
  Do not clean it up just put it up there and let others do it! I'd be very interested in it, please reach out when you do.
freediver 5y ago
Another robust solution is Tranquility reader which exists as an extension and has better accuracy than Readability at the expense of speed. https://github.com/ushnisha/tranquility-reader-webextensions https://github.com/ushnisha/tranquility-reader-webextensions
- Tarsul 5y ago
  yes! I use this addon for firefox all the time. Usually I click on my "Tranquility!" button the moment the cookie notification pops up, no cookies needed ;) (good for articles via HN, also possible to circumvent some paywalls)
guessmyname 5y ago
I ported Mozilla’s Readability library to Go a couple of years ago [1] and use it every day to power a custom RSS feed of Hacker News via Reeder [2]. This is not a novelty, many people have ported Readability to different programming languages over the years. [1] https://github.com/cixtor/readability https://github.com/cixtor/readability [2] https://github.com/cixtor/rssfeed https://github.com/cixtor/rssfeed
dredmorbius 5y ago
My suspicion is that there are an increasing number of publishers who are intentionally severing compatibility with Readability. Washington Post, I'm looking at you mofos. Chief reason I'll seek out any alternative news site for archival. It's been this way for about a year, if not more.
mft_ 5y ago
O/T, but thanks for Push To Kindle. I found the browser version so useful I bought the iPhone app - both to use and also to support you. Brought a whole new field of usefulness to my Kindle
- k1m 5y ago
  Thank you! That's really nice to hear. (Appreciate the support too.)
mdoms 5y ago
Just reading through that code it seems like readbility is only intended to work on English language websites? Like it checks for nodes with class names matching /and|article|body|column|content|main|shadow/ and uses a minimum length of 140 characters for matching nodes that are reader-able. Seems a bit lazy for a company whose stated mission is "to ensure the Internet is a global public resource, open and accessible to all".
- Vinnl 5y ago
  For what it's worth, even when working on Dutch websites my class names will usually be in English, and I think that's common across the industry. After all, the programming languages we use to write them are already in English, so Dutch class names would stick out like a sore thumb.

by k1m 5y ago

infogulch 5y ago

ArchiveBox is a tool that downloads web pages and saves them in various different formats: warc, pdf, rendered png, plain text. I wonder what it uses for plain text extraction and if the readability repo would be useful for that purpose. Edit: Oh neat it does actually. https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_readability https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#... > Archive method SAVE_READABILITY > Extract article text, summary, and byline using Mozilla's Readability library. Unlike the other methods, this does not download any additional files, so it's practically free from a disk usage perspective. It works by using any existing downloaded HTML version (e.g. wget, DOM dump, singlefile) and piping it into readability.

jrochkind1 5y ago
That's pretty amazing it already does it. ArchiveBox and the other stuff from the "DIY no-credentials don't-care-about-the-rules" web archiving community, like ArchiveTeam.... continues to astound me with it's quality and "professionalism" (as a credentialed professional in the field of digital library stuff... they are often outdoing the actual credentialed professional community).
- nikisweeting 5y ago
  Aww thank you!

up6w6 5y ago

I believe the Instant View[1] crowdsourcing model where people write templates for each website could boost a lot these parsers (hope they open source it soon). Its just impossible to make these extensions work for every single website with some simple heuristics. Check the codebase of some popular parsers: Firefox (already mentioned): https://github.com/mozilla/readability/blob/master/Readability.js https://github.com/mozilla/readability/blob/master/Readabili... Google Chrome: https://github.com/chromium/dom-distiller https://github.com/chromium/dom-distiller Mercury parser: https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser [1] https://instantview.telegram.org/ https://instantview.telegram.org/

k1m 5y ago
Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.
- freediver 5y ago
  Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?
  - k1m 5y ago
    In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
    - freediver 5y ago

jamil7 5y ago

I wrote a Swift port of it last year for my app but it deviates from readability a fair bit as I tailed it a bit to my needs, I've considered cleaning it up and open sourcing it regardless. I know there is an Objective-C port floating around.

freediver 5y ago
Do not clean it up just put it up there and let others do it! I'd be very interested in it, please reach out when you do.

freediver 5y ago

Another robust solution is Tranquility reader which exists as an extension and has better accuracy than Readability at the expense of speed. https://github.com/ushnisha/tranquility-reader-webextensions https://github.com/ushnisha/tranquility-reader-webextensions

Tarsul 5y ago
yes! I use this addon for firefox all the time. Usually I click on my "Tranquility!" button the moment the cookie notification pops up, no cookies needed ;) (good for articles via HN, also possible to circumvent some paywalls)

guessmyname 5y ago

I ported Mozilla’s Readability library to Go a couple of years ago [1] and use it every day to power a custom RSS feed of Hacker News via Reeder [2]. This is not a novelty, many people have ported Readability to different programming languages over the years. [1] https://github.com/cixtor/readability https://github.com/cixtor/readability [2] https://github.com/cixtor/rssfeed https://github.com/cixtor/rssfeed

dredmorbius 5y ago

My suspicion is that there are an increasing number of publishers who are intentionally severing compatibility with Readability. Washington Post, I'm looking at you mofos. Chief reason I'll seek out any alternative news site for archival. It's been this way for about a year, if not more.

mft_ 5y ago

O/T, but thanks for Push To Kindle. I found the browser version so useful I bought the iPhone app - both to use and also to support you. Brought a whole new field of usefulness to my Kindle

k1m 5y ago
Thank you! That's really nice to hear. (Appreciate the support too.)

mdoms 5y ago

Just reading through that code it seems like readbility is only intended to work on English language websites? Like it checks for nodes with class names matching /and|article|body|column|content|main|shadow/ and uses a minimum length of 140 characters for matching nodes that are reader-able. Seems a bit lazy for a company whose stated mission is "to ensure the Internet is a global public resource, open and accessible to all".

Vinnl 5y ago
For what it's worth, even when working on Dutch websites my class names will usually be in English, and I think that's common across the industry. After all, the programming languages we use to write them are already in English, so Dutch class names would stick out like a sore thumb.