>Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit”. yeah, the fact that it is actually useful for blocking crawlers is kind of a mis

>Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit”. yeah, the fact that it is actually useful for blocking crawlers is kind of a misleading thing. it's called "robots.txt", it's there to help the robots, not to block them. you use it to help a robot crawl your site more efficiently, and tell them what not to bother looking at so they don't waste their time. people seem to have forgotten really quickly that making your website as accessible as possible to crawlers was actually considered a good thing, and there was a whole industry around optimizing websites for search engines crawlers.