5 ms·

https://developers.cloudflare.com/bots/reference/verified-bots-policy/ https://developers.cloudflare.com/bots/reference/verified-bo... https://radar.cloudflare

by buzer 3y ago

https://developers.cloudflare.com/bots/reference/verified-bots-policy/ https://developers.cloudflare.com/bots/reference/verified-bo... https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots https://blog.cloudflare.com/friendly-bots/ https://blog.cloudflare.com/friendly-bots/ > At Cloudflare, we manually “verify” good bots, so they don’t get blocked.

gochi 3y ago
>and honor robots.txt.
NicoJuicy 3y ago
Nice links! But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt) Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers. It seems that cloudflare is actually fixing this problem and making competition for Google possible here. I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it) Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason-why-competing-with-google-is-so-hard https://www.fastcompany.com/90709672/the-little-known-reason... Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).