3 ms·

I tried to dig it out of the PHP source but without a local checkout it was non-obvious: I wonder if any such "synthetic rss feed" system honors the ETag and/or

I tried to dig it out of the PHP source but without a local checkout it was non-obvious: I wonder if any such "synthetic rss feed" system honors the ETag and/or Last-Modified and/or cache headers of the target page, or if every rss feed refresh unconditionally loads the upstream page only to throw away 90% of the generated html And that's not even getting into the raging tire fire that is akamai / cloudflare / whatever anti-bot technologies. I did see support for http proxies, but it wasn't clear if that was something one could set on a per-xpath-feed basis or whether the whole system had to run under one proxy (potentially $$$)

Nextgrid 3y ago
Proxies won't help - most of the aforementioned providers now do TCP, TLS and browser fingerprinting and use a heuristic approach. You need to be able to provide a consistent fingerprint to all of those to pass. Proxies are actually pretty useless, you need to go one layer lower to be able to fake those fingerprints, what you need is a VPN instead (by VPN I mean an IP-level tunnel, not a public VPN provider - those IPs are already blacklisted and often are so for good reason).