4 ms·

I believe the Instant View[1] crowdsourcing model where people write templates for each website could boost a lot these parsers (hope they open source it soon).

by up6w6 5y ago

I believe the Instant View[1] crowdsourcing model where people write templates for each website could boost a lot these parsers (hope they open source it soon). Its just impossible to make these extensions work for every single website with some simple heuristics. Check the codebase of some popular parsers: Firefox (already mentioned): https://github.com/mozilla/readability/blob/master/Readability.js https://github.com/mozilla/readability/blob/master/Readabili... Google Chrome: https://github.com/chromium/dom-distiller https://github.com/chromium/dom-distiller Mercury parser: https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser [1] https://instantview.telegram.org/ https://instantview.telegram.org/

k1m 5y ago
Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.
- freediver 5y ago
  Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?
  - k1m 5y ago
    In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
    - freediver 5y ago
      Makes sense. What does 'prune' and 'tidy' instruct parser to do?
      - k1m 5y ago
        Prune instructs the parser to remove any elements within the extracted article block that look superfluous. This can result in false positives, so we tend to disable it when we've gone to the trouble of creating site-specific extraction rules. Tidy determines if the source HTML should be cleaned up first with HTML Tidy - https://github.com/htacg/tidy-html5 https://github.com/htacg/tidy-html5. If you're parsing the source HTML with an HTML 5 parser, as we are now, it shouldn't be necessary any more (I think we actually ignore it now). We used it more before when we relied on libxml parsing, which often trips up on modern HTML.
- benzible 5y ago
  FYI your API pricing page doesn't seem to load https://rapidapi.com/user/fivefilters https://rapidapi.com/user/fivefilters
nyanpasu64 5y ago
Is it possible to utilize the database of Instant View per-site parsers in a web browser or extension's reader mode?
bspammer 5y ago
Another cool crowdsourced thing I discovered recently is SponsorBlock [1] which is an extension to automatically skip sponsored content in Youtube videos. Users contribute timings to the database that everyone else uses. It works remarkably well, any recent video with more than about 50,000 views is pretty much guaranteed to have timings submitted. [1] https://sponsor.ajay.app/ https://sponsor.ajay.app/

by up6w6 5y ago

k1m 5y ago

Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.

freediver 5y ago
Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?
- k1m 5y ago
  In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
  - freediver 5y ago
    Makes sense. What does 'prune' and 'tidy' instruct parser to do?
    - k1m 5y ago
      Prune instructs the parser to remove any elements within the extracted article block that look superfluous. This can result in false positives, so we tend to disable it when we've gone to the trouble of creating site-specific extraction rules. Tidy determines if the source HTML should be cleaned up first with HTML Tidy - https://github.com/htacg/tidy-html5 https://github.com/htacg/tidy-html5. If you're parsing the source HTML with an HTML 5 parser, as we are now, it shouldn't be necessary any more (I think we actually ignore it now). We used it more before when we relied on libxml parsing, which often trips up on modern HTML.
benzible 5y ago
FYI your API pricing page doesn't seem to load https://rapidapi.com/user/fivefilters https://rapidapi.com/user/fivefilters

nyanpasu64 5y ago

Is it possible to utilize the database of Instant View per-site parsers in a web browser or extension's reader mode?

bspammer 5y ago

Another cool crowdsourced thing I discovered recently is SponsorBlock [1] which is an extension to automatically skip sponsored content in Youtube videos. Users contribute timings to the database that everyone else uses. It works remarkably well, any recent video with more than about 50,000 views is pretty much guaranteed to have timings submitted. [1] https://sponsor.ajay.app/ https://sponsor.ajay.app/