3 ms·
Cloudflare is a large part of this, they actively block all crawlers except for Google.
by groffee 4y ago
Cloudflare is a large part of this, they actively block all crawlers except for Google.
- SheinhardtWigCo 4y agoThat seems like a strategic vulnerability for Google. Things will get interesting if CF goes into search.
- ColinHayhurst 4y agoNot true actually [0]. In my experience, at least, Cloudflare has been way more responsive than other big names that I won't call out here. [0] https://radar.cloudflare.com/verified-bots https://radar.cloudflare.com/verified-bots