3 ms·
Readability (https://github.com/luin/readability https://github.com/luin/readability) is a classic, and included as part of Firefox (I think, maybe that's been
by kough 10y ago
Readability (https://github.com/luin/readability https://github.com/luin/readability) is a classic, and included as part of Firefox (I think, maybe that's been discontinued). It's essentially a bag of hand-written heuristics but they're pretty good heuristics.
Some interesting reading is Christian Kohlschütter's thesis on this problem, which is framed in academia as "how do we assemble good text corpuses from webpages for data analysis, which means removing junk (boilerplate) from our HTML crawls" (https://code.google.com/archive/p/boilerpipe/wikis/WSDM2010Paper.wiki https://code.google.com/archive/p/boilerpipe/wikis/WSDM2010P...). Boilerpipe would probably be the right way to go, but if you're not using Java it could be harder to integrate.
- userhacker 10y agoI second Readability, it works great for article heavy webpages. I used it to build a reading time estimator for chrome https://chrome.google.com/webstore/detail/read-time/nccohhimobidpghgpnejnbkpoichbbml https://chrome.google.com/webstore/detail/read-time/nccohhim... and its open source https://github.com/usergit/read-time https://github.com/usergit/read-time bonus, you can click on the extension to show only the main content of the page
- cpeterso 10y agoFirefox maintains a fork of Readability for Firefox's reader mode here: https://github.com/mozilla/readability https://github.com/mozilla/readability