5 ms·

Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https

by k1m 5y ago

Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.

freediver 5y ago
Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?
- k1m 5y ago
  In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
  - freediver 5y ago
    Makes sense. What does 'prune' and 'tidy' instruct parser to do?
    - k1m 5y ago
      Prune instructs the parser to remove any elements within the extracted article block that look superfluous. This can result in false positives, so we tend to disable it when we've gone to the trouble of creating site-specific extraction rules. Tidy determines if the source HTML should be cleaned up first with HTML Tidy - https://github.com/htacg/tidy-html5 https://github.com/htacg/tidy-html5. If you're parsing the source HTML with an HTML 5 parser, as we are now, it shouldn't be necessary any more (I think we actually ignore it now). We used it more before when we relied on libxml parsing, which often trips up on modern HTML.
benzible 5y ago
FYI your API pricing page doesn't seem to load https://rapidapi.com/user/fivefilters https://rapidapi.com/user/fivefilters

by k1m 5y ago

freediver 5y ago

Thanks to both of you for expanding my 'readable web' toolbox. What do you fallback to if the rule is not present or doesn't work?

k1m 5y ago
In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.
- freediver 5y ago
  Makes sense. What does 'prune' and 'tidy' instruct parser to do?
  - k1m 5y ago
    Prune instructs the parser to remove any elements within the extracted article block that look superfluous. This can result in false positives, so we tend to disable it when we've gone to the trouble of creating site-specific extraction rules. Tidy determines if the source HTML should be cleaned up first with HTML Tidy - https://github.com/htacg/tidy-html5 https://github.com/htacg/tidy-html5. If you're parsing the source HTML with an HTML 5 parser, as we are now, it shouldn't be necessary any more (I think we actually ignore it now). We used it more before when we relied on libxml parsing, which often trips up on modern HTML.

benzible 5y ago

FYI your API pricing page doesn't seem to load https://rapidapi.com/user/fivefilters https://rapidapi.com/user/fivefilters