Any time I think about robots.txt, I think about a quote from Pirates of the Caribbean. [1] "The only rules that really matter are these: what a man can do and

Any time I think about robots.txt, I think about a quote from Pirates of the Caribbean. [1] "The only rules that really matter are these: what a man can do and what a man can't do." except that I replace man with bot. Everything should be designed to handle pirates, given the hostile nature of the internet. To me, robots.txt is a friendly way to say, "Hey bots, this is what I allow. Stay in these lanes including crawl-delay [2] and I won't block you." Step outside and I can put you on an exercise wheel. I know very few support crawl-delay since it is not part of the standard but that is not my problem. Blocking bots or making them waste a lot of cycles or get dummy data or wildly reordering packets or adding random packet loss or slowing them to 2KB/s is more fun for me than playing Doom. [1] - https://www.youtube.com/watch?v=B4zwh26kP8o https://www.youtube.com/watch?v=B4zwh26kP8o [video][2 mins] [2] - https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directi...