3 ms·

Our company site has a very strict anti-scraping Cloudflare mechanism. Will your crawler be able to work around it?

• sph 3y ago
  Which means that requests to your website might randomly return HTTP 403. At the moment, this is such a common issue I just ignore HTTP 403 errors when there's a Server: Cloudflare header (and a handful of others). That said, Cloudflare has a process to register as a "good bot" [1], which I believe we are (we respect robots.txt, keep a single connection alive per host, etc.), so it's possible that once I set that up, the gods that control the Internet might let us through. In general though, Bernard follows the HTTP spec but cannot do miracles. If your webserver is trigger-happy or very strict, it is your responsibility to let us through if you want to use the service. 1: https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots