3 ms·

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / Use

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Extended Disallow: / User-agent: Google-Extended-Factual Disallow: / # Block Microsoft's Search & AI Crawler User-agent: Bingbot Disallow: /

frereubu 2mo ago
This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#syntax https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.
- FinnKuhn 2mo ago
  They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though. https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...
spiderfarmer 2mo ago
90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.
- 6510 2mo ago
  I once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with <img> <map>'s
pixelesque 2mo ago
Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place. So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.
globular-toast 2mo ago
I won't eat your lunch if you put a sticker on your lunch box telling me not to.
Cthulhu_ 2mo ago
This only works for crawlers that play fair, unfortunately there's many that don't. Including ones that use consumer devices like TVs [0] and mobile phone apps that offer incentives to consumers (if they're even open about it) to use their internet connection to e.g. crawl websites. There's probably an army of hacked toothbrushes and the like doing the same thing. [0] https://news.ycombinator.com/item?id=49000864 https://news.ycombinator.com/item?id=49000864

frereubu 2mo ago

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#syntax https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.

FinnKuhn 2mo ago
They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though. https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...

spiderfarmer 2mo ago

90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.

6510 2mo ago
I once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with <img> <map>'s

pixelesque 2mo ago

Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place. So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.

globular-toast 2mo ago

I won't eat your lunch if you put a sticker on your lunch box telling me not to.

Cthulhu_ 2mo ago

This only works for crawlers that play fair, unfortunately there's many that don't. Including ones that use consumer devices like TVs [0] and mobile phone apps that offer incentives to consumers (if they're even open about it) to use their internet connection to e.g. crawl websites. There's probably an army of hacked toothbrushes and the like doing the same thing. [0] https://news.ycombinator.com/item?id=49000864 https://news.ycombinator.com/item?id=49000864