3 ms·
It’s a truly distinct and amazingly designed website that I’m often not smart enough to understand. It’s a unique property on the Net, and that’s saying a lot t
by tomcam 5y ago
It’s a truly distinct and amazingly designed website that I’m often not smart enough to understand. It’s a unique property on the Net, and that’s saying a lot these days.
One thing I would like to understand more is the archive system (used to reduce the problem of dead links). Is there some kind of automated process to create local copies of the linked webpages, or is it done by hand using some thing like wget or curl? How far do they go to archive embedded assets such as images or PDF files? Are there any copyright considerations for archived articles (what if you archive someone’s article that they would prefer to keep off-line)?
- voussoir 5y agoThere are plenty of rabbit holes to get into when it comes to web archiving, but for blog authors that just want to preserve a page referenced by their text, using a tool like SingleFile is a great first step. https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile
- tomcam 5y agoDef a good start, thanks.
- chefandy 5y agoWeb archiving goes pretty deep, actually. I have no idea what this site uses but the open source tools maintained by the Webrecorder project are rad.
- jeduardo 5y agoThere is some info about archiving here: https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs Lots of jobs and scripts, plus usage of archive.org as well. It's an interesting read.
- gwern 5y agoThat page is a bit outdated because I am still finetuning the on-site archive system before I do a writeup. I still use archiver-bot etc, they're just not how I do the on-site archives. See https://github.com/gwern/gwern.net/blob/master/build/LinkArchive.hs https://github.com/gwern/gwern.net/blob/master/build/LinkArc... https://github.com/gwern/gwern.net/blob/master/build/linkArchive.sh https://github.com/gwern/gwern.net/blob/master/build/linkArc... for that. The quick summary is that PDFs are automatically downloaded, hosted locally, and links rewritten to the local PDF; other URLs, after a delay, call the CLI version of https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile to run headless Chrome to dump a snapshot, which are manually reviewed by myself & improved as necessary, and then links get rewritten to the snapshot HTML. They get some no-crawl HTTP headers and robots.txt exclusions to try to reduce copyright trouble.
- tomcam 5y agoTHANK YOU for scratching that itch.
- tomcam 5y agoJust what the doctor ordered. Thank you.