3 ms·
On a vaguely related note, I hacked together a WayBack machine type thing on top of Common Crawl in like 50 lines of Python. So it transparently lets you load a
by andai 2y ago
On a vaguely related note, I hacked together a WayBack machine type thing on top of Common Crawl in like 50 lines of Python. So it transparently lets you load any page directly from the dumps (it loads only those few bytes from the crawl dumps, and gunzips them).
It was really easy, and I was surprised it apparently hadn't been done before.
The only problem is the index IP blocks you if you query it too often, and it gets queried once for every resource on the page.
I've been informed that there are ways around this (download an index?), does anyone know more about that?
- speerer 2y agoThis sounds a fantasic idea. Will you be releasing it?
- RestartKernel 2y agoCouldn't this still work as a service if you run all queries through the front-end?
- skylerwiernik 2y agoI imagine that they'd get blocked by CORS
- RestartKernel 2y agoGood point. CORS somehow didn't cross my mind.
- andai 2y agoThanks. I meant that light usage from a single user was already enough to get banned. (Though if you rewrite the HTML so there's no resources to fetch, it should be OK...) Also, it's just a proxy, there's no front-end. (Being a 50 line Python script with one dependency (Flask), the idea is that the end user would run it.) It's just http://localhost/URL http://localhost/URL, and it loads it from the hardcoded snapshot. I guess https://localhost/snapshot/url https://localhost/snapshot/url might work better. I'd also like to see how many lines it ends up being if you remove the one dependency. I've done this for other projects and the answer is "not much more" and "also runs way faster for some reason."
- RestartKernel 2y agoI see, thanks for the extra details. I'm pretty sure you can forgo Flask in favour of directly using the underlying WSGI protocol, while keeping the the Web server you are already using (Gunicorn, probably). The Web server would still be a dependency though.