3 ms·

I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it. Never assume malice what can be attributed to incompetence.

• cyanydeez 1y ago
  Its likely just a shitty attempt to rate limit bots
• aequitas 1y ago
  Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.