5 ms·

"When we detect unauthorized crawling..." How did you do that?

• mog_dev 2y ago
  Simple, you add the trapped paths to robots.txt Well behaved robots will not crawl them.
  • d4rkn0d3z 2y ago
    Nifty trick.
  • CaffeineLD50 2y ago
    And the misbehaved bots follow the path right into the pit and then...the Void of Infinite AI Abyss.
  • ccgreg 2y ago
    Cloudflare's documentation says that Labyrinth is not based on robots.txt.
    • Epskampie 2y ago
      In line 1 of of the linked page: "waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives".
      • ccgreg 2y ago
        Does that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.
• hombre_fatal 2y ago
  Just consider how you click around HN versus how your crawler would behave if you wanted to crawl every page of HN starting from the homepage.