3 ms·Cloudflare's documentation says that Labyrinth is not based on robots.txt.by ccgreg 2y agoCloudflare's documentation says that Labyrinth is not based on robots.txt.Epskampie 2y agoIn line 1 of of the linked page: "waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives".ccgreg 2y agoDoes that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.