4 ms·

Why the internet needs crawl neutrality

• groffee 4y ago
  Cloudflare is a large part of this, they actively block all crawlers except for Google.
  • SheinhardtWigCo 4y ago
    That seems like a strategic vulnerability for Google. Things will get interesting if CF goes into search.
  • ColinHayhurst 4y ago
    Not true actually [0]. In my experience, at least, Cloudflare has been way more responsive than other big names that I won't call out here. [0] https://radar.cloudflare.com/verified-bots https://radar.cloudflare.com/verified-bots