3 ms·
># Crawlers that are kind enough to obey, but which we'd rather not have ># unless they're feeding search engines. >[disallows UbiCrawler, DOC and zao] Anyon
by BbzzbB 5y ago
># Crawlers that are kind enough to obey, but which we'd rather not have
># unless they're feeding search engines.
>[disallows UbiCrawler, DOC and zao]
Anyone understand this bit? They are obedient spiders yet are disallowed because of purpose? Are these obedient but still overly intensive spiders?
- zinekeller 5y agoIt seems that all of them were built for recursive archiving in the early '00s for personal storage (so they were programs literally installed on computers rather than a specific group). Probably a legacy thing (considering at that time Wikipedia literally runs on a single server). I found a robots.txt file that even list a greater variety of said software: https://www.iatse.com/robots.txt https://www.iatse.com/robots.txt