4 ms·

Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when

Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore? Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare? Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?

dannyw 5y ago
There are a lot of hypotheticals here. I think you'll convince CloudFlare, and their customers, if you could name names and mention specific examples? If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
- jopsen 5y ago
  Or site owners may actively want to block a price comparison site.. Depending on the industry, etc..
  - thejosh 5y ago
    Agreed on this point, and most companies who would want to do these sorts of things would restrict it even if Cloudflare didn't exist. I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
- hysan 5y ago
  Price comparison example: https://shucks.top/ https://shucks.top/ sometimes gets blocked by cloudflare. Most recent was getting blocked from checking B&H.