4 ms·
I wrote something similar some years back for a bot I used to run. Given an input URL, it concurrently attempted to create snapshots across as many archive site
by mintplant 9y ago
I wrote something similar some years back for a bot I used to run. Given an input URL, it concurrently attempted to create snapshots across as many archive sites (archive.is, Wayback Machine, etc) as possible, with caching, retries, smart backoff, and continuous updating of the destination as HTTP responses arrived. Always meant to release it as a standalone library, but never got around to doing so. I wonder where the code is now...
- robryk 9y agoGwern has released such a piece of software: https://github.com/gwern/archiver-bot https://github.com/gwern/archiver-bot See also http://www.gwern.net/Archiving-URLs http://www.gwern.net/Archiving-URLs for a description of their usage of it.
- mintplant 9y agoSort of. Looks like it archives to each service serially, doesn't retrieve the resulting snapshot URL, and doesn't do any caching, retries, etc. If one service goes down, starts responding slowly, or blocks your requests, it'll break. This is like the first version of my own implementation; it took a lot more work from there to make it robust.