2 ms·

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#syntax https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.

FinnKuhn 3mo ago
They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though. https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...