I am not surprised... I am unfamiliar with web caching proxies like squid [0] but I am wondering if that might be the most straightforward way to do this. So

I am not surprised... I am unfamiliar with web caching proxies like squid [0] but I am wondering if that might be the most straightforward way to do this. So use squid and then have a batch job that go through /var/spool/squid every day and update your web archive according to some defined filters. - [0] https://www.squid-cache.org/ https://www.squid-cache.org/