4 ms·
Well Cloudflare doesn’t even block Google’s AI crawlers because they don’t differentiate themselves from their search crawlers. Cloudflare gives Google an unfai
by rzz3 1y ago
Well Cloudflare doesn’t even block Google’s AI crawlers because they don’t differentiate themselves from their search crawlers. Cloudflare gives Google an unfair competitive advantage.
- warkdarrior 1y agoGoogle claims their AI crawlers have user agents distinct from the search crawlers: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/...