4 ms·

There is some info about archiving here: https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs Lots of jobs and scripts, plus usage of arch

There is some info about archiving here: https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs Lots of jobs and scripts, plus usage of archive.org as well. It's an interesting read.

gwern 5y ago
That page is a bit outdated because I am still finetuning the on-site archive system before I do a writeup. I still use archiver-bot etc, they're just not how I do the on-site archives. See https://github.com/gwern/gwern.net/blob/master/build/LinkArchive.hs https://github.com/gwern/gwern.net/blob/master/build/LinkArc... https://github.com/gwern/gwern.net/blob/master/build/linkArchive.sh https://github.com/gwern/gwern.net/blob/master/build/linkArc... for that. The quick summary is that PDFs are automatically downloaded, hosted locally, and links rewritten to the local PDF; other URLs, after a delay, call the CLI version of https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile to run headless Chrome to dump a snapshot, which are manually reviewed by myself & improved as necessary, and then links get rewritten to the snapshot HTML. They get some no-crawl HTTP headers and robots.txt exclusions to try to reduce copyright trouble.
- tomcam 5y ago
  THANK YOU for scratching that itch.
tomcam 5y ago
Just what the doctor ordered. Thank you.