3 ms·
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers li
by ccgreg 3mo ago
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
- sumedh 3mo agoCloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
- m463 3mo agocloudflare routinely denies my human-piloted browser now, on many sites.
- ambigious7777 3mo agoyou're not a bot thats irrelevant
- pocksuppet 3mo agoI wonder if you'd have more reliable internet browsing with a browser that was a CF verified bot.
- pineapplepizza6 3mo ago[dead]