2 ms·
blekko attempted to implement crawling a local copy, and it was a PITA. We'd rather crawl the real thing with a crawl-delay of 1. Best would be if the Wikimedia
by greglindahl 13y ago
blekko attempted to implement crawling a local copy, and it was a PITA. We'd rather crawl the real thing with a crawl-delay of 1. Best would be if the Wikimedia Foundation made a .html dump available.