3 ms·
A different kind of robots.txt: this is annotated on why they're blocked: https://meta.wikimedia.org/robots.txt https://meta.wikimedia.org/robots.txt
by zinekeller 5y ago
A different kind of robots.txt: this is annotated on why they're blocked: https://meta.wikimedia.org/robots.txt https://meta.wikimedia.org/robots.txt
- npteljes 5y agoThis is fantastic. The wget one even has instruction in how to play nice.
- renewiltord 5y ago> /wiki/Requests_for_comment/Rodrigo_Tetsuo_Argenton@brwikimedia Everyone Loves Rodrigo
- BbzzbB 5y ago># Crawlers that are kind enough to obey, but which we'd rather not have ># unless they're feeding search engines. >[disallows UbiCrawler, DOC and zao] Anyone understand this bit? They are obedient spiders yet are disallowed because of purpose? Are these obedient but still overly intensive spiders?
- zinekeller 5y agoIt seems that all of them were built for recursive archiving in the early '00s for personal storage (so they were programs literally installed on computers rather than a specific group). Probably a legacy thing (considering at that time Wikipedia literally runs on a single server). I found a robots.txt file that even list a greater variety of said software: https://www.iatse.com/robots.txt https://www.iatse.com/robots.txt