4 ms·
One way that Cloudflare is gatekeeping is by declaring which bots are AI Bots. Common Crawl's CCBot is used for a lot of stuff -- it's an archive, there are mor
by ccgreg 1y ago
One way that Cloudflare is gatekeeping is by declaring which bots are AI Bots. Common Crawl's CCBot is used for a lot of stuff -- it's an archive, there are more than 10,000 research papers citing common crawl, mostly not AI -- but Cloudflare deems CCBot to be an "AI Bot", and I suspect most website owners don't have any idea what the list of AI Bots is and how they were chosen.
- lyu07282 1y agowould be an obvious loophole if you could just use CC instead of paying cloudflare no?
- ccgreg 1y agoIt's a similar loophole as public libraries. When I was a kid, I read thousands of books from the library, without paying anyone anything. But as for the crawl loophole: CCBot obeys robots.txt, and CCBot also preserves all robots.txt and REPL signals so that downstream users can find out if a website intended to block them at crawl time.