3 ms·
https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy Disclaimer - It's a project of mine, specifically written for my in
by fake-name 8y ago
https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy
Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready.
It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1].
It does both rewritten archiving (basically overwriting the page style) and raw archiving.
I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet.
I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already.
1: https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever
2: https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController
3: https://github.com/fake-name/WebRequest https://github.com/fake-name/WebRequest