4 ms·
I doubt they'd add these protections for visiting a handful of links at human speed. Correct me if I'm wrong, but crawlers often send hundreds of requests per m
by compootr 2y ago
I doubt they'd add these protections for visiting a handful of links at human speed. Correct me if I'm wrong, but crawlers often send hundreds of requests per minute, testing random outlinks and routes they find on the site
My concern would be as a webmaster: serving useless content to users, and as a user: not getting the information from the site.
I probably wouldn't use this feature, since I often deploy static websites that use little to no resources, and the potential harm outweighs the benefit
- ccgreg 2y agoCloudflare blocks Common Crawl's bot no matter how slow it goes.