5 ms·
Is there open source software that can extract the "content" part of a given page cleanly? I'm thinking about what the reader mode in browsers can do as an exam
by throwawaysea 5y ago
Is there open source software that can extract the "content" part of a given page cleanly? I'm thinking about what the reader mode in browsers can do as an example, where the main content is somehow isolated and displayed.
- specproc 5y agoI believe the main library for reader mode is called readability. I played around with a python implementation a while back. Just pipe in your raw html as part of the process. It's good, but not flawless. If I remember correctly, it included some quotes and image text as part of the body for the site I tried it on.
- stef25 5y agoThere's a PHP port of Readability and it works for some sites, for others not at all. Very far from perfect.
- tannhaeuser 5y agoYou can use SGML (on which HTML is/was based) and my LGPL-licensed sgmljs package [1] for that, plus my SGML DTD grammar for HTML5. [2] describes common tasks in preservation of Web content to give you a flavor, but you can customize what SGML does with your markup to death really; in your case, you'll probably want to throw away divs and navs to get clean semantic HTML which you can do using SGML link processes (= pipeline of markup filters and transformations), but you could also convert HTML into canonical markup (eg XML) and use Turing-complete XML processing tools such as XSLT as described in the linked tutorial. [1]: http://sgmljs.net http://sgmljs.net [2]: http://sgmljs.net/docs/parsing-html-tutorial/parsing-html-tutorial.html http://sgmljs.net/docs/parsing-html-tutorial/parsing-html-tu...