2 ms·

To be fair, I would love a curated version of Wikipedia periodically exported as a self contained reference application. I thought this would be a good use of t

To be fair, I would love a curated version of Wikipedia periodically exported as a self contained reference application. I thought this would be a good use of the Internet Archive capabilities too. At Blekko we, of course, crawled all of Wikipedia. One of the more interesting aspects was how many dead links it had to references (which presumably at one time were not dead). Sometimes you could find those references in the wayback machine and sometimes they were just "poof" gone. This is the nature of the web, information isn't persistent. Hence I think periodically pulling it into static storage would be a good thing for longevity.

duskwuff 3y ago
> One of the more interesting aspects was how many dead links it had to references (which presumably at one time were not dead). Sometimes you could find those references in the wayback machine and sometimes they were just "poof" gone. This is largely a solved problem. There's a number of bots on Wikipedia and other WMF wikis which periodically trigger Internet Archive dumps for all external links which are used as references, and which can replace those references with links to the archive if a site goes offline.