3 ms·
Would a WARC format reduce effort needed to make Reader-like programs? Ie strip pages of HTML cruft, leaving you with text, images, etc - the content?
by lijogdfljk 4y ago
Would a WARC format reduce effort needed to make Reader-like programs? Ie strip pages of HTML cruft, leaving you with text, images, etc - the content?
- uniqueuid 4y agoNot at all. A WARC file gives you exactly what you would see on the wire, or in the network inspector tab of your browser. It does nothing to the content, and that's the point. The only thing you gain (and that's very important for other reasons as well) is an immutable ground truth to work from when creating the reader view of a given article.
- lijogdfljk 4y agoGotcha - yea i was hoping maybe it snapshotted the HTML or some such, side stepping some issues long dynamic text or JS shenanigans
- simonw 4y agoI've found the Readability.js library to be really good for that - here's my recipe for running it as a CLI: https://til.simonwillison.net/shot-scraper/readability https://til.simonwillison.net/shot-scraper/readability