2 ms·
Ask HN: Why Is Internet Archive Included in Cloudflare's List of AI Crawlers?
Was just looking into some steps to block AI bots from crawling a site, a feature that Cloudflare promoted recently and noticed on their list of "Verified Bots" for AI Crawlers, one called "Internet Archive" is included. Why?
There is also another one "InternetArchiveBot" listed in a different category as "Monitoring & Analytics". Will blocking the AI one mess with Wayback Machine activities and that kind of 'useful' thing?
The recent announcement post: Block AI bots, scrapers and crawlers with a single click https://news.ycombinator.com/item?id=40865627
https://radar.cloudflare.com/traffic/verified-bots
- ChrisArchitect 2y agoLinks as above: The recent announcement post: Block AI bots, scrapers and crawlers with a single click https://news.ycombinator.com/item?id=40865627 https://news.ycombinator.com/item?id=40865627 https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots